- The line between functional programming and OOP is way grayer than most silly online articles try to make it seem. If you have mostly fixed data, and you think your project will scale by adding functionality, then functional programming is a good fit. If most of the functionality is already modelled and fixed, and you think your project will scale by adding more data, then OOP is a good fit. Most projects are combinations of both.
- Supercontest
- A good software design reminder: separate functions that read/write data from those that analyze the data. For example: the function that calculates if a pick is good. It queries the picks and matchups, does the math, then commits the results. Instead, it would be better as two functions: one that does the math, then a wrapper that does the query, calls the math func, then does the commit. This makes it a lot easier to unittest. You can write one just for the analytical part, then another that tests the db interfaces.
- Webtesting options.
- flaskr. http://flask.pocoo.org/docs/1.0/testing/.
- flask-webtest. https://flask-webtest.readthedocs.io/en/latest/.
- flask-testing. https://pythonhosted.org/Flask-Testing/.
- They're all pretty similar. Can create a client app connected to a db, get/post all routes, login/logout, context and sessions, sqlachemy support, json responses, selenium, etc.
- I'm gonna use flask-testing.
- The dev containers (flask and postgres) only consume about 10% memory (maybe 300MB). Not much cpu diff.
- Unittests.
- Changed get_app to accept a URI to make unittesting easier (creating a temp postgres db).
- Remember the javascript analogs for python:
- python=node, pip=npm, virtualenv=nodeenv, pylint=eslint, pytest=jasmine
- Ran some delinting for javascript.
- Did a little more bazel research. Still not the most useful thing for python, which doesn't need to be built. Think of it like a make system. You have inputs, you have recipes that define what to do with the build, and then you have outputs (usually files).
- The real power comes when everything in a highly complex system is built with bazel, because then you can define the deptree for the whole project/monorepo/whatever, even across languages.
- Everything is a file, remember! This is why I think the python-package-support is a halfway solution.
- Keanu Reeves founded a motorcycle company in 2011 named ARCH and it is headquartered in Hawthorne, CA!
- Started using the whatsapp web app on my laptop. Convenient.
- Also switched my default messaging (SMS) app to google, which has a web app. Texting from my browser on the laptop!
- Changed the trackpad setting from area to fingers (1 anywhere for click, 2 anywhere for right click).
- Reinstalled GNOME system monitor. Wasn't working properly. Still amazed at how resource-hoggy chrome is.
Tuesday, April 16, 2019
Monday, April 15, 2019
- Supercontest
- Ansible
- Falls back on paramiko (python) if openssh fails.
- Setup
- sudo apt install ansible (this gives ansible-playbook and others)
- added southbaysupercontest.com to /etc/ansible/hosts
- ssh-agent bash
- ssh-add ~/.ssh/digitalocean
- ansible all -a "/bin/echo hello"
- A lot of ansible's power used to be the composition of various microservices, which docker-compose is a much simpler alternative for. It still has relevance in fleet management, however, and that's what I'll use it for.
- Refamiliarized myself with their documentation.
- I could just write a shell script to ssh, git pull, run the make target for docker-compose, then rsync dump files back and forth, but I'm gonna use ansible just for fun. It will enable scaling, load balancing, and more later (if need be).
- Added playbooks to deploy the whole app remotely from local, as well as backup and restore the db. You simple enter through a oneliner make call, which farms out to ansible (which goes back and calls other make recipes).
- Closed ticket #10.
- Did a few test deploy/backup/restore operations between my laptop and the production server.
- Upgraded docker-compose to 1.24 on southbaysupercontest.com. The py apt version was installed.
- Unittests
- Looks like there's some nginx testing capability: https://github.com/nginx/unit/tree/master/test.
- The module for a temp postgres db is: https://pypi.org/project/testing.postgresql/.
- There are some docker unittest frameworks as well, but because it's a generic tools, the test capabilities are generically shallow: https://github.com/GoogleContainerTools/container-structure-test.
- Chromedriver
- Tested the linefetching capability in the docker container. There were holes.
- libnss3 was necessary for chromedriver.
- You also need the chrome binary. Added.
- Couldn't verify end-to-end without some spoofing, because westgate has no lines during the offseason.
- Here is another awesome script: https://gist.github.com/ziadoz/3e8ab7e944d02fe872c3454d17af31a5. If I have any problems at the start of the 2019 season, I can just add that to app/Dockerfile.
- You can't band-aid a poor engineering decision with software: https://www.youtube.com/watch?v=H2tuKiiznsY.
- To rebalance panes in tmux so they're equally spaced, used ctrl-b then alt-1 (vertical) or alt-2 (horizontal).
- Python 3 natively keeps command history across interpreter sessions (like ipython does). Very convenient.
Sunday, April 14, 2019
- BBQ
- The smoker went for 8.5hrs unattended last night. When I woke up, the coal basket and water pan each had about 30% remaining? At 175 and you could go 10+ hrs. Great to know. Turned it hot and spritzed, will transfer to the oven in about an hour.
- Supercontest
- Python 3.
- iteritems -> items
- operator.itemgetter -> dict.get
- dict.keys(), dict.values(), and zip() all return iterators now, not iterables. Cast to list when necessary. This is nice for memory saving, obviously.
- import pdb; pdb.set_trace() -> breakpoint()
- asyncio, obv
- docker-compose restart <> is the same as start -> stop. Useful for reloading.
- But there's an even better way to iterate in development. Added --py-autoreload=1 to the uwsgi cmd call in the app container (same as flask's dev server reload). Now when my files change (based on the mounted volume of my source), the uwsgi server reloads. Great!
- Watch the app in realtime with docker-compose logs -f <> !
- This confirms the app restart on change as well.
- Effectively about 2.7k lines in the entire infrastructure of my application:
- find -type f -not -path "*.venv*" -not -path "*.git*" -not -path "*.idea*" -not -path "*backups*" -not -path "*egg-info*" -not -path "*.pyc*" -exec wc -l {} +
- 2743 total
- There's only 800 lines restricting it to actual application code (python extensions within the supercontest/ subdir):
- find supercontest -name "*.py" -type f -not -path "*.venv*" -not -path "*.git*" -not -path "*.idea*" -not -path "*backups*" -not -path "*egg-info*" -not -path "*.pyc*" -exec wc -l {} +
- 864 total
- Pretty amazing. Under 1000 lines of custom code to produce a fully functional website.
- Adding all the enterprise bells and whistles of dev, management, deployment, etc only adds about 2000 more lines.
- Furthermore, the infra is twice the size of the source. Not surprising.
- Ran pylint (via tox) for the first time in a longggg time.
- Most common: no-member on all the usages of `db` (to commit, add, delete, etc).
- Closed #19.
- To search with ag but only for a specific filename regex (like an extension), use:
- ag -G "\.ext" keyword
- JS. Class inheritance, templating. Prototypal inheritance (from other objects), more like composition. Composition is almost always favorable.
- Angular has more of a pattern of two-way binding (frontend object updates, backend does as well, and vice versa). React is more one-way dataflow. The model is the only source of truth. The view just reflects it.
- Game of thrones final season premiere!
- My typing cursor had been jumping around a bit. It's a little annoying. The browser typing isn't as consequential, but it's dangerous in vim when I'm actually writing code. The active line changes based on autocomplete and other things, and I don't want any unintended visual shift to slip under the radar because of this.
- I changed the cursor theme back to adwaita (default) in hopes this would help, based on this answer: https://askubuntu.com/questions/115349/why-does-my-cursor-jump-when-typing. Don't think it's gonna change much. Changing the movement speed just means you can't move to a new line as quickly, which is a bandaid on the problem. A real solution would be changing the sensitivity. Maybe disabled touch clicks (and only registering push-clicks) would help. Otherwise, I just have to be very cognizant about my thumbs not touching the pad when typing (the setting for "disable touchpad when typing" is already on, but doesn't work well).
- Ya, that worked. Disabling tap-to-click means that errant touches don't misplace my cursor. Well worth the trade of not being able to silently click on something.
Saturday, April 13, 2019
- Supercontest.
- Continued work on the ever-helpful makefile. Added backup-db and restore-db. pg_dump defaults to output a sql text file, which pg_restore cannot read (lol). psql can, tho. Instead, I chose to have pg_dump output a custom file (which pg_restore CAN read, along with dirs and tars).
- If just performing alterations, then you can connect to the db and modify as you'd like. If doing a DROP -> CREATE (which can include other things like schema and role changes), then there obviously can't be other active connections to the db. Therefore, the flask container must be down before a restore runs.
- Made the services and the containers the same name for convenience.
- Got backup/restore working, but then it failed on a completely fresh postgres image because pg_dump does not include roles. Postgres keeps these at the cluster layer, which applies to all databases. I'll need to change it to pg_dumpall.
- Ended up actually not having to backup and restore the roles. Instead, I added docker/database/public.conf env_var back to the postgres image so that initdb would create the supercontest db and the supercontest role. Then my restoration just uses it to fill the data.
- alter [table/database] [matchup/public.user/alembic_version/pick] owner to supercontest
- drop role bmahlstedt
- Added a healthcheck to the database container startup to make sure 5432 was ready. (pg_restore was running before postgres was actually up). JK, just added a sleep after the docker-compose up.
- Ran some interesting queries on the data. Adding all teams picked by all players for the 2018 season, the top-picked teams were (1) chiefs (2) pats (3) rams. Shows that there's subconscious bias for better teams, even when the lines are supposedly 50/50.
- In a makefile, VAR=value doesn't actually set it until used (lazy). VAR:=value forces it to evaluate and set upon declaration.
- I'd been using docker-compose up/down, which removes and creates the containers as well. If you just want to start and stop the existing containers, use start/stop.
- Dad's bday. Spoke on the phone for a while.
- Started the big bbq for tomorrow. Brisket rubbed and put in the smoker overnight, lowest temp (175). Targeting 10pm-6am bark formation, then foil and oven.
Friday, April 12, 2019
- HN/MD.
- Read a bit about GraphQL again.
- A curried function takes multiple arguments and processes/returns one at a time, sequentially.
- Pretty awesome (but decently long) walkthrough of what a db is dong under the hood: https://cstack.github.io/db_tutorial/.
- Amazon obviously listens to everything Alexa hears (anonymously). https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio.
- Agile Lite. A more flexible, but still structure approach to sprinting: https://github.com/davebs/AgileLite.
- Fascinating statistical snapshot of the whole software community: https://insights.stackoverflow.com/survey/2019.
- Tested out a new stylesheet: https://github.com/kognise/water.css. Was pretty cool, but I kept my old CSS. This (and other just-add-css options) would be nice for future changes, to mix it up for the users.
- Investments.
- Nowadays, it's not just a seed and venture round then an IPO 6 years later. There are many growth rounds, and companies can stay private for longer than a decade, continuing to increase value. Companies stay private longer because it makes the investors more money.
- Founders and VCs have preferred stock. They have RSAs instead of ISOs. They can create liquidity events whenever they want with another growth round. They incur a lot less risk, while owning a substantially larger share of equity. Employees are just kinda along for the ride (which is not crazy, since employment is opt-in).
- The liquidity goalposts have moved in the founder/VC favor, but the vesting goalposts for employees have not. Many believe that there should no longer be a vesting cliff because its original timeline was basically to finish at the IPO. Now, that's no longer the case.
- If a company does offer liquidity events for employees regularly, these arguments go away.
- Dilution is real.
- https://steveblank.com/2019/04/10/startup-stock-options-why-a-good-deal-has-gone-bad
- Something weird happened with my docker-compose installation.
- Ran apt install python3-apt because it was missing the apt_pkg. Strange.
- Tried pip installing some stuff. ln -sf /usr/bin/pip3 /usr/bin/pip
- Cat /usr/bin/docker-compose. No, docker-compose shouldn't be a python entry script.
- Ahhhh. docker-compose is installed from git, not apt. The package on apt is a python package. https://docs.docker.com/compose/install/. Reinstalled normally.
- Picked up, trimmed, and brined 2 brisket packers and 2 pork butts for Sunday's GoT party.
- sudo apt autoremove killed a bunch of stuff again. I'm not going to run it anymore.
- Some important packages: sudo apt install ubuntu-software update-manager ubuntu-desktop gdm3 gnome-control-center
- Ugh aaaand it removed the display manager. Unbelievable.
- Stuck in a login loop. ctrl-alt-f3 to enter terminal.
- First, ubuntu is the dist based on debian. It's the OS.
- On top of ubuntu, there are desktop environments. Unity has been the primary for a long time, but now the default is Gnome for 18. Another option is Xfce (Xubuntu).
- In addition, there are display managers (login managers). gdm is the default gnome one, and lightdm is a common alternative.
- I had to reset everything back to gnome manually because autoremove removed it. Ridiculous.
- `wmctrl -m` shows the desktop manager (window manager). Mine is back to gnome now.
- `cat /etc/X11/default-display-manager` shows the login manager. Mine is back to gdm3 now.
- A few more vim improvements.
- Started using tabs instead of split windows. Just ctrl-t (like chrome) then a filename to open a new one. ctrl-w to close. Then \ and left or right arrows to move between them.
- @ at the beginning of a command in a makefile makes it not echo the command itself. Unlike bash, it does print by default.
Thursday, April 11, 2019
- Updated default python to 3.7 (was 2.7) - `ln -sf /usr/bin/python3.7 /usr/bin/python`
- Reconfigured the weekly backups to basically include my home dir but skip all subdirs. Ran my first backup.
- More on the black hole image from yesterday; they stitched together a virtual aperture much larger than the individual imagers in order to achieve higher resolution, exactly like our work at Stanford with atmospheric lensing! The lead spoke about the process in a Ted Talk 2 years ago: https://www.youtube.com/watch?v=BIvezCVcsYs. It's the same. We did some lunar photography by stitching together images (pixels) that refracted through various eddies in the atmosphere, creating a virtual aperture of a few kilometers. Their black hole target was much farther away. The simple diffraction limit required an aperture of about the size of the Earth (thousands of kilometers) in order to get the resolution to see the black hole so far away. So, the algorithm stitched together a TON of data from telescopes around the globe, over time, so the rotation of the Earth itself caused the observatories to cover more of the surface area of this virtual aperture. Same exact process. Reached out to Ved to revel.
- Alt Shift X honestly has 50+ GoT videos, each 10-15m. I watched most on 2x speed.
- The old Grandmaester Pycelle had a weird close relationship with Tywin.
- Qyburn reanimated The Mountain. Cleganebowl would be soooo sick.
- Lady Stoneheart is the reanimated Catelyn Stark.
- The Valyrian empire held most of Essos until the Doom. All secrets about how to make Valyrian steel were lost, most of the dragons died, and the city of Valyria is basically nuclear fallout. The Targaryens were the only survivors because they left before. The Doom is the antithesis to the Long Night, the fire and ice contrast of the series.
- The Arm of Dorne was a land bridge that the First Men used to cross from Essos to Westeros. It was destroyed by the Children of the Forest's gods so no more colonizers would come.
- The Horn of Winter brings down the wall and raises giants. It might be in the possession of Sam Tarly.
- Placed Amazon Fresh order.
- Removed two .desktop files in ~/.local/share/applications that were lingering but not installed. This was creating false icons in the applications menu (one for google play music and one for google remote desktop).
- Added all common apps to dock (favorites) and cleaned the desktop background a bit.
- Created symlink from ipython to ipython3.
- Messed with my tmux conf a little.
- Finally set up ctags (and uninstalled pycharm, the only real thing I used it for).
- sudo apt install exuberant-ctags
- Added tag location to vimrc `
- Indexed the tags: `ctags -R -o ~/.tags $(~/code/supercontest/.venv/bin/python -c "import os, sys; print(' '.join('{}'.format(d) for d in sys.path if os.path.isdir(d)))")`
- Takes about 4s and generates a tag file of about 20MB for supercontest.
- Added a bash script to wrap this with a venv, then an alias to wrap that so you can run it whenever.
- ctrl-] to jump to, ctrl-o to jump back
- Here is a neat way to reindex automatically with git: https://tbaggery.com/2011/08/08/effortless-ctags-with-git.html. The most common way is probably to reindex on a schedule with a cron job.
- Added dockerignore (for the .venv right now)
- Had to reinstall python3.7-dev to build the C extensions for psycopg and uwsgi.
- To reload a config file, run `source .config_file`
- Remember, the most portable shebang is `#!/usr/bin/env bash`
- PW/MD
- TDD = test driven development.
- Thrust profiles usually decrease through a trajectory. This is seen commonly if you blow up a balloon and let it go. See that little kick at the end when it seems to go faster? F=ma. As mass decreases, and force stays constant, acceleration must increase. In order to keep the vehicle from speeding up, subjecting it to higher load stress, thrust can be decreased.
- Vim
- :Explore to open netrw (directory browser)
- Added Vundle to my vimrc. Remember to call :PluginInstall after any changes.
- Fugitive has a ton of super useful git integrations. You can blame, search history, checkout a file in mem, etc. I chose NOT to install this. I like keeping my text editing separate from my version control interface.
- Use vim for multiple windows (instead of tmux). Use tmux for multiple processes. For multiple files being edited at once, use vim. This is a bit of shift from what I'm used to, even years into my career.
- :sp file/path/blah to open a new file in a horizontal split. :vsp for vertical.
- ctrl-w then an arrow to move between windows, just like ctrl-a for tmux.
- Maximize is ctrl-w then _, go back with ctrl-w then =
- :h or :h <plugin> for any help
- Added the syntastic plugin for pylint/flake8 as well, but then removed it. Needs an active flake8/pylint on PATH, which I don't wanna mess with since I develop in docker. Just ended up using the built-in python syntax highlighting. I've got enough experience with Python to not need realtime lint notification.
- Added the Jedi plugin for autocompletion. ctrl-space to open autocompletion.
- Summary:
- You have ctags now: ctrl-] and ctrl-o
- Split windows in vim instead of tmux: :sp file/path and ctrl-w arrow
- Autocompletion: ctrl-space
- Syntax highlighting.
- The vimrc improved a ton of things for tabs, wrapping, color, etc.
Wednesday, April 10, 2019
- TJ Dillashaw suspended two years for EPO! That's the blood doping that Armstrong was involved with. It's a severe performance enhancer for cardio. Brendan Schaub said before this breaking news that USADA had found a better way to screen for EPO, and fighters were going to start getting popped: https://www.youtube.com/watch?v=NABFMFHnstw&feature=youtu.be&t=3963. Amazing. Cody Garbrandt also specifically called him out for EPO before this.
- Cleaned kitchen and organized utensils into a compartmental drawer finally.
- Ran the SSL Labs tests against the encryption I set up for southbaysupercontest.com:
- Got an A: https://www.ssllabs.com/ssltest/analyze.html?d=southbaysupercontest.com
- Here are the averages: https://www.ssllabs.com/ssl-pulse/
- You can deduct state and local taxes (SALT) from federal taxes, up to the max (right now it's 10k). TurboTax accommodates this, but you have to itemize. Worth looking at if you're paying a lot in state taxes.
- In docker-compose yamls, 'expose' means the port is exposed for linked services (like flask<->nginx) but not to the host computer. 'ports' are for exposing to the host.
- Letsencrypt limits to 5 cert requests for the same domain per week.
- FW.
- Docker has the .dockerignore file available, just like git. This can be used to skip files/directories from COPY, etc.
- Use apt-get and apt-cache in scripts instead of apt. Much more parseable output.
- First ever image of a black hole! Good description here (and it preceded the image!) - https://www.youtube.com/watch?v=zUyH3XhpLTo.
- Watched a tonnnn of alt shift x game of thrones videos to prepare for the premiere of the final season.
- I always forget the history of the north. The first men came and attacked the natives, the children of the forest. The children needed help, so they captured a man and stuck dragonglass in his heart. This turned him into the first white walker, the night king. Eventually, the children and men came to a pact. Fast forward, and the army of the dead has grown uncontrollably, and it's men+children vs them. This war was called the long night, or war for the dawn. The dead lost and were pushed back. Bran the builder (the first stark) and the children helped build the wall. The night's watch was created to keep them north. Wildings then became a thing. They don't have a historical significance like the dead/children/firstmen, they're just free folk in the risky north.
- While that account is the most historical, and is common in the north (called "old gods" with nature, weirwood, ec), there are other religious beliefs of the history. King's landing obvious believes in the seven (Andals), hence the hesitance to believe in the white walkers. Melisandre believes in the lord of light, R'hllor, who use a sword called lightbringer. A prophecy says he'll be reincarnated as the prince that was promised, who Melisandre believes to be Jon Snow.
- The Andals came much later. They are from Essos and invaded Westeros, becoming the main culture we see in the show, with the exception of the north (they didn't invade that far).
- Wargs can possess animals. Bran is a warg. Bran also has greensight (visionary time travelling dreams).
- White walkers are the captains. The hordes they convert are wights.
- Dragonglass is obsidian. It is different than Valyrian steel, which is manmade. These two are the only materials that kill white walkers (+wights).
- The three eyed raven is just a human. He was hand of the king back in the targaryan days, when his name was bloodraven, then left to become lord commander of the night's watch, then disappeared north of the wall. He went to a cave at the base of a weirwood tree and basically mated his body with the trunk. He's the last of the greenseers and can use visions to communicate (called greensight). This is how he finds and teaches Bran, ultimately bringing him to the cave. Then the night king killed him.
- Supercontest.
- Docker/letsencrypt
- Remember, now that my uwsgi file is listening on a socket :8080, it can't serve http directly. It needs to be wrapped with a web server. To run in develop mode locally, you need to forward to a tcp socket and specify the protocol as http.
- I enabled this by rearranging the compose and uwsgi configurations. It's now modular for prod and dev.
- A service (or dockerfile) can only have one CMD. A RUN is called when the image is built. A CMD is something that's called every time the container starts. You can have multiple RUN calls. The state of the container after they've finished will be built into the image.
- Oh man, I spent a while trying to debug my services. It was just that LetsEncrypt was down, so my requests for certs (even dummy certs from staging) were getting ReadTimeout errors. You can check status here: https://letsencrypt.status.io/.
- https://letsdebug.net/ is a great site for troubleshooting why you might be failing letsencrypt http challenges.
- Another great site: https://transparencyreport.google.com/https/certificates. You can publicly view the certificates that have been granted to any site! Useful for debugging your own.
- Certbot can either run an HTTP challenge or a DNS one to verify ownership of domain.
- After a very long and frustrating (but educational!) sequence of troubleshooting, I made some progress. `docker-compose exec container_name command with no quotes` yielded some interesting info. My nginx container had an old conf file at /etc/nginx/conf.d. In all of my iterations on init-letsencrypt, the web_server container hadn't been rebuilt to serve the webroot files at /var/www/certbot, so the challenges were failing when they requested them. Rebuild your images when you change anything in their dockerfiles!
- This allowed my changes to ingest, isolating the problems faster. I rewrote nginx.conf with clearer simplicity, and removed the 'www.southbaysupercontest.com' from all entries. It's just specified in the certbot command line call, not the nginx conf anymore.
- Upgraded to Python 3.7 finallyyyyy
- sudo apt list --installed *python*
- sudo apt remove <the python packages you want>
- You can accidentally remove wayyyy too much if you autoremove *python*
- sudo apt update && sudo apt upgrade
- sudo apt install python3.7
- The service brought up and the main page was functional on the very first docker-compose up with the 3.7 python image, no problems! I'll do more advanced digging soon.
- Accidentally messed my machine up during some uninstalls. Would only boot into a terminal. Ended up being a reinstall of ubuntu-desktop.
- Enabled automatic backups of my home folder on Google Drive.
- `apt list --installed | wc -l` is 1782!
- gnome-tweak is the package that offers extended settings (like removing the trash icon from the desktop).
Tuesday, April 9, 2019
- Started the brisket. Went on at 845am. I'm getting the process down pretty smoothly now.
- Listened to big bootie mix 15 for the first time.
- Submitted my purchase offer election and emailed the subsequent transmittal form. Requested 900.
- Ahhhh, this is what our dodgeball team is named after: https://www.youtube.com/watch?v=cVsQLlk-T0s.
- Went to del amo to pick up the bar end that they forgot to replace originally. Took it home and installed it myself, just a #5 billet assembly for the bar end that extends about a half inch for the #3 mirror assembly to clamp onto.
- You cannot use letsencrypt for local development. You cannot certify that you own an IP address, and you cannot certify that you own the generic "localhost" domain. Usually you can just provide yourself a self-signed certificate to make sure https works before you deploy to production.
- Many benchmarks of the original py3 release were slower than py2. That's no longer the case, they're close for most (or advantage 3).
- Worked an Autotest issue with briley, pointing toward the hyperion.shared.install_autotest module to show how to provision python workers with monorepo packages (to avoid conflicts with first-parties being used as third-parties).
- Supercontest
- Commented out the `COPY dump.sql -> docker-initd` in the database Dockerfile. If expects a db in the mounted volume, it shouldn't rebuild an old one everytime. If you want to restore another database, do it manually.
- Shifted the development vs production workflows. Dev will just use runserver and flask with a simple sqlite database, no containers, no nginx, no postgres, no https.
- Set up automatic certification. Adds a 4th container, certbot, which exposes its certificates to the nginx container over a shared docker volume. It then runs certbot renew and nginx reload every few hours to ensure that they never expire.
- Because the current (whole) source directory is mounted, you CAN just make a change to the app on the host (on the droplet) and it will be reflected immediately in the app container (because the supercontest package is installed editably).
- Deployed to production
- Followed the instructions (I wrote) in ADMIN.md. Installed docker, created the secrets files, ran the cert initializer, then started the services.
- Copied the db backup over (from my laptop): scp -i ~/.ssh/digitalocean code/supercontest/backups/postgres/dump.sql southbaysupercontest.com:~
- Restored the data (on droplet): cat ~/dump.sql | docker exec -i postgres psql -U postgres
- Got https green, commited and pushed, closed ticket #27.
- Fully done with all 4 docker containers, a wonderful composition.
Monday, April 8, 2019
- Bought a whole packer. Prime. ~9lbs untrimmed. $3.50/lb (snow crab was almost $10/lb!). Trimmed and brined it. Will smoke brisket tomorrow. Separated the point and the flat.
- My pitmasterIQ arrived as well! Pretty easy setup. Ran a power cable out of the window, connected the fan to the bottom vent, and snaked the RTD (with alligator clip!) to the top grate. Will use it tomorrow with the brisket.
- Shaved head for the first time in (years?)! Easy, free, clean. Better for motorcycle helmet, gym, summer.
- Watched GoT recaps for seasons 1-7.
- `sudo -i` is used to enter a root shell. This is how you do certain things as root, like cd. Since you can't `sudo cd <>` you have to change to root (you could also do this with su) then cd there in the root shell.
- Docker volumes.
- Docker mounts are stored on the host in /var/lib/docker/volumes by default, if you don't specify a host location. This is useful if you don't need host files to be mirrored to the container, you just want the data in the container to persist when the container is destroyed. This is common for databases.
- If you specify a hostLocation:containerLocation volume, then it's basically a bidirectional mount point. This is useful if you want changes on the host to sync into changes on the container, and vice versa. This is common for source (git repos).
- You can share volumes across containers, btw, because they're all persisted on the host.
- You can do `docker volume ls` just like images or containers. You can `docker inspect <volume_name>` as well.
- If volumes were created by docker-compose, you should delete them with docker-compose as well: docker-compose down --volumes (this will delete your data!)
- You only need to pass `--build` to `docker-compose up -d` if..... you want to rebuild the images before starting the containers! If you don't, just run UP. It will be way faster.
- Supercontest.
- Test. Changed my picks. docker-compose down, then back up. The picks were gone, as expected, because I didn't set up the volume for them yet. Added the volume, then reperformed the test. They persisted.
- Got all volumes working as expected.
- The 3 main containers and docker-compose should be in a good place now. Moving on to the letsencrypt container.
- NCAA national championship.
- Last dodgeball game of the season. Playoffs begin next week.
Sunday, April 7, 2019
- Supercontest docker work.
- Only the db image reads the postgres config information in docker-compose. Instead of passing it via env variables to the flask image so that it can connect to the db as well, the flask image directly reads the config files. This means that the app can be started without docker, in the usual ways I'm used to (runserver, etc).
- Started testing the full system with docker-compose up.
- docker system prune -af
- If containers ever can't be stopped even with -f (due to apparmor, or other), run
- sudo killall docker-containerd-shim
- docker-compose down
- Successfully got `docker-compose up --build -d` to work. Just need to debug the actual application now.
- To open a shell in the nginx container: docker exec -it nginx /bin/bash
- Most base images on the official docker registry will pipe their main logs to stdout/stderr, so you can simply view them with `docker logs <containername>`.
- For example, /var/log/nginx/access.log -> /dev/stdout, so you can do `docker logs nginx`
- Eventually ran `sudo systemctl stop apparmor && sudo systemctl disable apparmor` so I didn't have to deal with the container removal errors anymore.
- Backed up the existing db just in case
- pg_dumpall -l supercontest > backups/dump.sql
- Stopped postgres on my host machine so the container could use the default port (5432).
- sudo service postgresql stop
- sudo update-rc.d postgresql disable
- Stopped nginx on my host machine so the container could use the default port (80).
- sudo servicectl stop nginx
- sudo servicectl disable nginx
- Added a postgres Dockerfile instead of just relying on the docker-compose yaml. Copied my dump to the /docker-entrypoint-initdbd/ folder so the existing database gets create when the container starts.
- To open psql in the container, just run `docker exec -it postgres psql`
- To perform a dev cycle, you can't just modify the code locally then rerun `docker-compose up --build -d`. It does use a cache, but because the container has already been created, and files have already been copied, it doesn't think it needs to do anything. What you really need to do is mount the source directory (if you're copying the whole over). If you're building a python package, just copying that over is fine.
- Resolvconf. To make the nameserver changes permanent (they get overwritten by the system).
- sudo apt install resolvconf
- sudo mkdir /etc/resolvconf/resolv.conf.d
- sudo vim /etc/resolvconf/resolv.conf.d/base
- nameserver 8.8.8.8
- nameserver 8.8.4.4
- sudo resolvconf -u
- netstat -na | grep <port_number>
- 8.8.8.8 and 8.8.4.4 are Google's public DNS servers (for IPv4).
- Remember, websockets are a protocol wrapped around tcp. They're very similar to http. Regular sockets are more generic, they're just tcp (usually).
Saturday, April 6, 2019
- Ryan Cabrera and O-Town performed at Saint Rocke last night, the bar literally 2 blocks up from my house, so a bunch of us went.
- Purchase offer. Created a few sheets and read lots of info on the details.
- I've found no great solutions for streaming from ubuntu to roku.
- Supercontest docker work.
- Added nameserver 8.8.8.8 and 8.8.4.4. to /etc/resolv.conf again.
- sudo docker run -it python:2.7 /bin/bash
- This command fetches the container (remotely, at first), then shells into it and enters a bash prompt.
- Made it so I don't have to type sudo every time. Just add yourself to the docker group.
- sudo gpasswd -a $USER docker
- newgrp docker # refreshes this login within your current shell
- Basically got all three containers and supporting files into a good place. Did test builds of a few dev images. Just need to test the actual application in container space now.
Friday, April 5, 2019
- Supercontest
- Postgres (#19)
- Changed the postgres 'supercontest' user's password to match the mail password, all inject into the app.config at the same time during initialization.
- In PyCharm's db navigator, my actual tables are under schemas -> public -> tables.
- Ahhhhhhhhhh. It did migrate my users properly. I was selecting the literal user table from postgres, which just had me. `select * from public.user` shows all 42 supercontest users. "public" is the schema which you have to run the query against.
- The leaderboard view works again. The select has to match the group_by in postgres. Postgres also has `DISTINCT ON`, which is useful.
- Fixed the week ordering issue. Just had to add .order_by(Matchup.week) to a couple queries. It looks like distinct() is what does the reordering. group_by() and all() and filter() didn't affect any of the existing order of the returns.
- Created a new alembic version, even though there had been no changes in the model yet. Alembic seemed to persist through the underlying sqlite->postgres migration without any issue. This makes sense, since it transacts with the db and manages it purely with SQL, independent of the DB implementation, but I'm still surprised that there were no gotchas here. Maybe there will be. I'm still suspicious.
- Alembic correctly autodetected the implementation context. It has a PostgresqlImpl class.
- Postgres has the notion of "sequences" which are commonly used as the primary key IDs for tables. You have to be explicit about this by defining SQLAlchemy().Sequence() in the schema (next to integer and primary key).
- Empty revisions (without --autogenerate) can be created by just changing 'migrate' to 'revision' like `python manage.py db revision -m "blah"`.
- Alembic --autogenerate cannot recognize new sequences. You have to do it manually. See my commit for the revision file: https://github.com/brianmahlstedt/supercontest/commit/18c44ed1b714c4058c4731f671ff27d59c37a85f#diff-a88a2de8356c14b4956f44589d249540
- PRed and merged the postgres branch after testing locally. Did not push to production server. I will wait until I implement the db container (next) to make that a bit easier to handle (revert/change).
- Docker
- sudo apt install docker.io docker-compose
- HN. Ya, common mispractices in hiring: http://tonsky.me/blog/hiring/.
- You can actually run linux containers on a windows host (but not vice versa) because newer windows machines ship with a base image of ubuntu in addition to the windows os.
- Enabled "Career Interests" on LinkedIn which lets recruiters know that I'm looking. Also provided my information to gain access to salary insights.
- Remember the command to compare your current branch to master: `git diff origin/master...`
Thursday, April 4, 2019
- Was able to get the facial verification process started for Binance. Should receive an email within a few days, then I can link my Binance account to cointracker, export a csv with all transactions, attach it to turbotax, file taxes, and delete the binance app from my phone.
- Nice, actually got verification later today. Added an API key for cointracker to my binance account, so I should have full tax info now.
- In order to get the forms necessary to input to turbotax, you have to buy the cointracker tax plan. This is bullshit. It's $49. I did it for convenience, but that's an insane fee.
- Total capital loss was about $6.2k. The max you can claim in a single tax year is 3k, so this will last me two years (2019's is called a capital loss carryover).
- Final submission of taxes!
- Another new thing: royalty distribution from my mesabi trust cbi shares. Reported on 1099-MISC, had to claim on Schedule E. Was only $42.
- Changed my medium digest back to daily. I just skim over the weekly and never look at anything. The daily would spark something cool every morning.
- My confluence space at SpaceX is actually saved. The user account is inactive, but all of the content is preserved: blog, pages, comments, everything.
- PW
- New versions of pycharm and django.
- Remember, Selenium can do more than just load dynamic content (js) into the dom for you. It can simulate the clicking of buttons, text input, scrolling, downloading, changing cookies, etc.
- sh returns 127 when it can't find the executable on path.
- systemctl communicates with systemd over dbus.
- Great collection of flask+nginx dockerfiles: https://github.com/tiangolo/uwsgi-nginx-flask-docker.
- Bottled this kombucha brew for second fermentation, finally. Used apple cider. The final brew time was 6 weeks, about 3x as long as usual. The cultures were super old and took forever, but the pH is ~2.5 and it's slightly tart. I added a little extra apple juice to bring the sweetness back up. Second fermentation should go in the fridge on Monday afternoon, 4 days is fine.
- Supercontest.
- Docker caches previous build results so you can step through a dockerfile as you develop on it.
- Read a lot about docker-compose and the final config of server/app/db in 3 separate containers. Gonna upgrade to postgres and python 3 first, then come back to that. Then enable backups.
- Changed the password for the southbaysupercontest@gmail.com account. It was getting correct-password attempts from elsewhere. It's now stored in my personal Google account.
- You can migrate a database to another type. Often, it means "dumping" the current db into a single file of pure SQL commands, which most databases' CLI allows you to do. Then you can import it into another database.
- This is also how backups often work. You just recreate the db from a dumped file.
- There are tools to perform this migration for you. pgloader is the main one to convert into Postgres, which is what I used to migrate the Supercontest db from sqlite.
- Upgraded to postgres (#19). Didn't finish, but got quite far.
- sudo apt install postgresql postgresql-contrib postgresql-client postgresql-client-common libpq-dev pgloader
- pip install psycopg2 (inside the sc venv)
- Added `export PGDATABASE=supercontest` to ~/.bashrc so that psql would by default connect to supercontest. To ingest, restart bash by simply typing bash.
- sudo -u postgres createuser --username=supercontest --superuser --pwprompt # run the command as the postgres user to create the auth that the app uses
- sudo -u postgres createdb supercontest # create the new db. dropdb is opposite
- pgloader ./data/supercontest.db postgresql:///supercontest
- Helpful commands inside psql:
- \c <> # connect to that database
- \l # list all available databases
- \q # exits this shell
- \conninfo shows data
- show data_directory; # lists where the db is actually stored
- select * from <>;
Wednesday, April 3, 2019
- To turn your stock into gravy, just add flour (about as much as the fat layer on top), whisk until smooth, then bring to a boil, then turn off. Did this with the chicken stock from yesterday. Turned out delicious.
- Didn't add any fat, there's enough in the poultry remains.
- This is effectively just smoked carcass, water, and flour (+ any vegetables/spices you add to make the stock).
- The bones yield more marrow and flavor once cooked, plus you get some smokiness.
- Don't wash raw chicken. The water splash can carry bacteria a few feet, all over surfaces and utensils and you!
- Taxes/crypto
- Coinbase account recovery completed, so I was able to get that transaction history.
- Started the account recovery process for Binance, because I still need those transactions.
- Created an account with cointracker.io. Linked Coinbase and Coinbase Pro (formerly Gdax). Once I get access to my Binance account again, I can link it to get the final numbers.
- Watched Wayne's World for the first time. I don't get it. The movie was terrible.
- Some of my favorite bands are all British: Beatles, Zeppelin, Queen, Pink Floyd, Rolling Stones, Elton John, Coldplay, David Bowie, Iron Maiden, The Who, Black Sabbath, Oasis!
- Reactivated amazon fresh.
- 8 organic vine tomatoes, 2 organic banana bunches, 8 heads of organic kale, 1lb greek yogurt, 1 jar peanut butter.
- $44 at Vons, $38 on Fresh.
- This is one week, so 4 orders per month. Fresh costs $15/month, so add $4 to each order. Still cheaper.
- And I don't have to ride the motorcycle to the store. Will order next week!
Tuesday, April 2, 2019
- Medi-Cal can take up to 45 days to respond to the application (which they accept year-round). Check digital and regular mail. If accepted, they cover medical bills up to 3 months before the application was received, so even though I applied before April, if they don't respond for another month, I can get retro coverage.
- Smoked chicken halves! Amazing.
- Also made chicken stock. Next time I want to try it with the smoked carcass in addition to the neck/backbone/giblets/wingtips.
- Remember, the full wing is drummette -> wingette -> wingtip. The wingette is the stereotypical chicken wing, like you'd receive at Buffalo Wild Wings.
- Aleksandr Karelin is the greatest wrestler of all time.
- Jeremy asked about backfilling previous seasons. I showed the script that does it autonomously. If Petty has old spreadsheets, and I add a "season" dropdown or year capability, this wouldn't be too hard.
- Called Spectrum to cancel our plan (now out of promotion) so that we get new promotion for half the price (back down to ~$45 for 100MB). Of course you can't do it online. The operator was nice. The service will cancel at the end of the cycle I already paid for, Apr 17th. Briley will pick up the next one.
Monday, April 1, 2019
- Paid rent. Next month it raises to 1100 (each).
- Taxes. Took about 3 hours. The usual W2, 1099-DIV/INT/MISC, VLF for motorcycle registration credits (3). No clothing donations this year. Had a 1099-B due to a merger that forced a sale of some shares (cap gains). Also crypto losses. Learned that you don't have to pay taxes on ESPP shares until they're sold.
- Did a ton of bbq research.
- Use the charcoal basket for better airflow. Load wood chunks into the bottom first, then charcoal, then light on top. Add pellets later for extra.
- Use a blower to get higher temps. This makes each charcoal load last a lot longer as well. Ordered one of these.
- Made the supercontest app public again.
- Looked up residential solar options. They're a few thousand up front, but give you (annual) tax breaks and reduce utilities (obviously).
- Researched different home automation technologies.
- Bought birthday presents for Eric and Dad.
- $80 Urthbox (3mo + 1 box free) subscription.
- $80 Philips home hub + 1 multicolor bulb.
- Interestingly enough, because of California's higher efficiency/green standards, lightbulbs are more expensive to buy on Amazon when shipping to a CA address.
- Messaged Kelsey/Jess/Derek on LinkedIn to hear their comparisons between SpaceX and Netflix.
- Remember, systemctl enable <service> means that the service will start on boot.
- If there are specific commands you want a specific user (e.g. you) to be able to run without having to type sudo every time, you can add that to the sudoers file or add yourself to that command's group (if it's configured that way).
- "Tweaks" is the name of the ubuntu package which extends settings. I disabled workspaces.
- Docker refreshers.
- By default, docker only shows currently-running info (container ls, etc). To check all (including history), use -a.
- Base images are like `ubuntu` or busybox. Literally just that. With my Dockerfile, I create a child image (from that) which runs additional commands and installations and stuff that customize the image for supercontest hosting.
- Different containers for database, app, etc. Docker compose pulls all these together in a multi-container app.
- Docker build is takes your Dockerfile and creates an image from it. Then you can run that local image.
- `docker container prune`
- Started working on #27 for the supercontest, dockerizing deployment.
- Just like at SpaceX, dns resolution was a pain with Docker. Ended up with the same solution, adding `nameserver 8.8.8.8` and 8.8.4.4 in /etc/resolv.conf.
Saturday, March 30, 2019
- Withdrew cash from my Robinhood account as well. Backing to checking. Only about $600.
- Watched a ton of franklin bbq videos.
- Supercontest. Added line graph for cumulative scores. PRed and closed #37.
- Charts.js is a standard clientside choice. You just pass the data structures to JS and it can plot. I chose to do server-side with plotly.
- Removed week 18 from the leaderboard views (table and graph).
- Deployed to prod. Told Greg.
- Defaulted to show only the current user's trace. They can still select others.
- Finished C++ tutorial on sololearn, google doc with notes as usual. I love this language. I love how explicit you have to be in declaration: datatypes, memory allocation, etc.
- Covered California replied and said that I might be eligible for Medi-Cal, and the office in my count will contact me soon.
Friday, March 29, 2019
- Lyft IPOed at $72 (for a total valuation of 20b) and opened at $87.
- All of my limit sales in Ally executed, so I withdrew the rest to BOA. The only remaining securities I have in Ally are Tesla/Microsoft/Apple (about 10k total).
- Securities by definition are the superset, including stocks, bonds, and others. Shares are units of stock.
- HN/M
- AMP = Accelerated Mobile Pages. Fast and mobile-supported.
- Just reminders. Let for block scoped. Const for unchanging. Var for execution-context scoped.
- Submitted the application for Covered California and Medical. I should qualify, since the termination makes me eligible outside of open enrollment (Fall) and current income is 0. This will be a lot cheaper than COBRA ($439/mo for just healthcare, dental is $42 more and vision is $7 more.
- If the application is approved, my enrollment/coverage begins 5/1.
- Their web interface was not great. I had to submit multiple times. I hope everything got through, as I made sure it did.
- Connected Lumin, a decent online PDF editor, to my Google Drive account.
- Swift tutorial on sololearn. Added doc to Google Drive.
- More march madnesss. All s16 games done.
- Collected feedback from folks for the supercontest. Added some color to the lb and all-user pick views, and made a ticket for the remaining items: https://github.com/brianmahlstedt/supercontest/issues/37.
Thursday, March 28, 2019
- Added up all utilities (so calif edison) and internet (time warner) for the last 24 months with briley, which includes through the March 2019 bills. Check gmail for the total. It's not settled yet, he's gonna cover internet for a while and we'll actually settle these 2 years later.
- FW/PW.
- Softball.
- March madnesssssssssss. Sweet sixteen.
- Emailed the supercontest app out to the sbsc18 players for feedback.
Wednesday, March 27, 2019
- Organized the final remaining prep items before interviews start! Moved from Google Keep to here, for posts in the next few days.
- Making lists of monthly costs for awareness:
- $1075 Rent
- $450 Food
- $150 Car Insurance
- $80 Cell Phone
- $40 Utilities
- $40 Internet
- $20 Gas
- $15 Amex
- $5 Mobile app subscriptions (Spotify, Calm, etc)
- Almost $2000/month for basics (no bars, no shopping, no trips). Right now, I have a few more out-of-pocket expenses that an employer would usually cover ($250 for health insurance and $40 for gym).
- Historically, I pay much more than this (about double!) due to restaurants, Amazon, and more. I'll certainly be a bit more observant.
- About half of my Ally limit sales have been met and fulfilled. Transferred 5k from my investment account to BOA. As I reinvest this money later, I will do so through Robinhood instead of Ally.
- Set my Mint account back up. I will start regularly checking this as the aggregator. Got familiar with it again. Feels good.
- Added https://raw.githubusercontent.com/synthead/mint-adfree/master/mint-adfree.txt to the uBlock filter list so the Mint dashboard is adfree and skips all the cards I don't wanna see.
- Scrubbed my Chrome account and removed the majority of saved passwords (SpaceX, old sites, college accounts).
- Watched Candyman, an old (92) horror classic I had never seen.
- Smoked ribs. Incredible.
Friday, March 22, 2019
- Instead of adding `style="display: none;"` to an element, you can just add `hidden`. It's a global html attribute, like class or id.
- IIS (Internet Information Services) is a Windows server.
- Ember, Glimmer, and Aurelia are alternative js framework to react/vue.
- D3 is a very popular JS framework for visualizing data. Graphs and much more.
- Lodash (and Underscore) are utilities libraries for JS that offer convenient functions like map and filter. As native javascript expands with similar built-ins, these become less and less necessary.
- Lottttts of March Madness in the background today, love it.
- Liquidated about 7k in personal investments to give me a little buffer as I conclude this sabbatical. Exelixis and Take Two Software were my cows, each returning over 400%. My basis years ago was a few hundred for each. Now they're both thousands. Left about 10k across Apple/Microsoft/Tesla, trending well still. Small transactions just to move enough around right now with a low market to give me some security. SpaceX is long overdue a purchase offer.
- HN/PW/M. Skimmed some stuff this week, but nothing grabbed my attention for a deepdive.
Wednesday, March 20, 2019
- Did TONS of Seattle trip prep. Weird places to go, cool sights, fun tours, museums, ghost stuff, car, hotel! It was $750 for flights/car/hotel for 4d/3n. Not bad. We'll see how much is added for food/tickets/etc.
- Groceries and cleaned the whole house: kitchen, bathroom, vacuum, more.
- Dinner with Alex and Spencer. I miss my dudes! We built a great product in a tough environment.
Tuesday, March 19, 2019
- Twitch is the external chat client you can embed. I always forget the name so I'm typing it here again.
- Gbro pooped on the carpet again. The standard pace was too fast, so about 6 weeks ago I slowed it down considerably. I duct taped the center hole fully, then opened it up about half a centimeter every week. Even this proved too much - we're in the 1% here! Holes make cats forget years of bathroom recognition. Giving up and going back to the dome.
- Did some research in Traeger and Yoder smokers.
- Went to Bludso's bbq in Hollywood, only a few blocks from Allie's house. Supposedly has some of the best BBQ in LA. I got 1/4lb brisket point, a chicken quarter of dark meat, and 3 spare pork ribs. Overall, I give it a 7. The chicken was the best. The brisket wasn't tender enough for a point, and the bark was underformed. Very small smoke ring. The ribs had great flavor but could have been silkier off the bone. They probably didn't crutch.
- Filled out two march madness brackets, one for a blake street league and one with the spacex dudes.
- You can use the <label for="id" ... to make the text on an input checkbox clickable. You can also do it with javascript click events, obviously.
- SELinux is security-enhanced linux. It's an add-on suite of tools to add security, like disabling network access, etc. It's disabled by default.
- Besides nginx logs and other app targets, you can always check /var/log/syslog. I forget the obvious sometimes.
- UFW BLOCK in your syslog is ufw blocking a connection, as it's supposed to. UFW is uncomplicated firewall, ubuntu's built-in.
- Remember that journalctl also has logs, different from the ones in /var/log.
- If you type `curl icanhazip.com` you can get your ipv6 address at the command line. Pass -4 for ipv4.
- If you want to empty a file but keep its location and permissions, run `cat /dev/null > /file/you/want/to/wipe-out`.
- SC
- Added winner col to the matchup table and points col to the pick table. Migrated and upgraded.
- Added leaderboard view! Updated layout, etc.
- Created a ticket to make the matchups and picks views piggyback on the serverside calculation of winners, rather than doing it clientside: https://github.com/brianmahlstedt/supercontest/issues/33.
- Wrote Python functions to parse Petty's excel master. One adds (registers) all the users to the User table. The other scrapes all the sheets for each week and adds the picks.
- Deleted myself and the example users before adding all the 2018 players.
- Remember, bad transactions will require a rollback() before the same session can be used properly again.
- Had to do some reverse engineering to figure out how Petty handled duplicate submissions (eg. 5 picks twice) and >5 pick single submissions (he chose the last five, not the first five). This won't be a problem for the app since I enforce it properly, but in order to be accurate in backreference, I made them match.
- PR, merged, closed ticket #30. Closed the MVP milestone! Moving on to some final items in the next milestone before sending it out for beta.
- Researched and closed most of the remaining tickets in the "before 2019 season" milestone.
- Got familiar with flask-assets and flask-cache, for bundling static files and for making expensive routes only run once every N time periods.
- Spent HOURS trying to resolve issues on the production server. ERR_CONNECTION_REFUSED, site can't be reached. Thought it was nginx, uwsgi, some of my source changes, DNS, digitalocean, reboot, the chrome cache clientside, much more. Checked all logs and many other attempted solutions. Ultimately was certs. Ugh. Just run `sudo certbot --nginx -d southbaysupercontest.com -d www.southbaysupercontest.com` and request/renew new certs (choosing 2 for the https redirect as usual).
- Some other changes on the production server: Installed xlrd into the venv (which you still have to do with any env changes -_-) and symlinked the service files (which you don't have to do anymore, the links will stay up to date now). Docker will make all this a lot easier.
- Sent the app to the guys in whatsapp for limited beta testing before forwarding to the whole sbsc email chain.
Monday, March 18, 2019
- SC
- Moved the js files around so they're only loaded on the pages that need them.
- Did the same for css files. Much easier to keep the templates/stylesheets/scripts separate for their various views, consistently named and applied.
- Added function for get_team_abv for the 3-letter code.
- Styled the all-picks table. Vertical text was nontrivial.
- My repo has functional structure currently - an alternative design is divisional structure, where there's a subdir for each route and the static files (templates/js/css) are all alongside it.
- Closed the blueprint ticket for the next milestone, doing it for the week/picks view nesting. Learned quite a bit here. Reorganized for blueprints.
- Added a url_value_preprocessor to determine requested_week and available_weeks before passing that information on to the routes in the week blueprint that need it. You're basically just taking it out of the native 'values' dict and putting it into the g object for your own use. Then you don't have to type week<week> in all the routes, and add the week arg to all functions, etc.
- Added url_defaults as well so that it queries the db for the most recent week on the site's homepage.
- Organized all the weekX/picks subpages, and added the links for them.
- Created ticket for the "don't show all picks till sunday" enforcement, to be done at the beginning of next year.
- Started the LB work.
Sunday, March 17, 2019
- The template extensions are relative to the template root dir, not local (you have to prepend main/ etc).
- You can use a <th> tag in place of <td> to get header formatting outside of the header row, like for a left bold column.
- Stack overflow was offline tonight for maintenance (<1hr).
- .replace is JS is only for the first occurrence. It's dumb. To replace all, change the first string (the one to change from) "example" to /example/g. You have to remove the quotes, which introduces further dumb requirements. If your string contains special characters, you need to escape them. For example, to remove all periods from an email, you'd do email.replace(/\./g, '').
- jquery selectors for ID can't have special characters like periods and such.
- SC
- Worked on #30, the pick view. Moved templates around to generalize the week header dropdown.
- Got the pick view basically done. I was surprised how nontrivial it was to handle the table. One of the biggest takeaways: give IDs to all cells in the table, named after the variables (x and y axis headers) passed through jinja. Then jquery can select them easily, rather than trying to find them on the fly.
- Remaining: some styling, and then generic abstraction of the static content. I basically had views/templates/js for the main table of a week's matchups, but now i need to separate all that for another table so that each has the appropriate scope.
Saturday, March 16, 2019
- Flask servers persist (or restart after) ubuntu suspension, fyi. If you close your laptop with the app running, it will continue when you reopen.
- You can have branches in alembic (just like git) for when multiple people are working on changes that modify the models. I accidentally created a branchpoint a while back on the production server. To resolve, you merge the versions (just like git). This creates a new version and migration script (mine was empty, as expected). Then you can migrate and upgrade as usual.
- For future reference - my net worth exceeded 1m at around age 28. Vested and exercised. Probably age 26 unexercised. The biggest hindrance? Taxes on illiquid assets. AMT traps are real. I paid hundreds of thousands of dollars before even seeing a cent of it. IRC 83(i) will help considerably, but the finance department shucked this novelty due to a clause where purchase offers make employees ineligible. Even...when the purchase offers are inconsistent (nondeterministic schedule), limited (you are severely capped in share sales), unknown (you have no idea when your trap hits), and infrequent (gaps longer than a year apart).
- The pycharm db browser is buggy if you start manually modifying data.
- HN
- Ya, the cookies warnings need to stop.
- "communicate in the language of options, not demands or preferences" https://news.ycombinator.com/item?id=19349676. I agree. This applies outside the workplace as well. Social issues like racism and other political debates largely stem from personal bias. A withdrawn, impartial view of options and their pros/cons would serve everyone well.
- This is really cool - basically a sed alternative for find/replace, but with machine learning. https://github.com/Inventitech/strans.
- The median age of Google/Amazon employees (software in general) is 30. The median age of american workers is 42.
- Error handlers are pretty easy in Flask. If the exception (or other code, or custom handler) is to be reached in an ajax call, however, you must do something different.
- SC
- Removed the backref from the picks table (to users). Left the FK constraint.
- Removed the points col. This will be a separate table.
- Added utilities.add_user for easy creation of new users (for testing). This obviously must be invoked from a db context (python manage.py shell), not just any old python console.
- Created and few users and made sure they were distinct.
- Moved some test dbs to backups. Merged an old db version, migrated, then upgraded my local dev db.
- Played with moving html <script> tags around to the proper template to avoid Undefined Variable errors.
- Did the ajax/flask error handling. If any of the common pick mistakes occur serverside (>5, started game, etc), it will return the traceback and flash the client! All are grouped in the InvalidPicks exception class within the view.
- Added clientside verification of "game already started" and "can only pick wed-sat". Changed the allowable pick back from status=F to P (the proper one, pregame).
- Closed ticket #4, created #30 for the last items in the mvp: the pick view and the leaderboard view (then back population from last season's data).
- Moved the logic for coloring to js (from html). Only applies to picks now, not all.
- Added an 18th week to the db so that people can play with the pick interface before next season.
Thursday, March 14, 2019
- PW
- Watched another tutorial on hidden API manipulation. You really can do some deep inspection with just chrome alone. Adding x-csrf-token headers, scraping for routes, etc. Remember, just run a single get in the same session to grab the token before adding it to headers in subsequent requests. https://www.youtube.com/watch?v=6zge0N962aw.
- Single page applications, server side rendering, and nuxt.js (for universal vue apps) - https://scotch.io/tutorials/building-a-universal-application-with-nuxtjs-and-django. next.js is React's framework for this.
- Few small improvements to my bbq game. Ordered a spice dispenser for even rubs, as well as 24" butcher paper for single-sheet wrapping (it was leaking through my separate 12" sheets). Also going to try using the coffee grinder next time for pepper - the hand crank is way too slow.
- You can access jinja template variables within javascript if you use the html rendering as a proxy. Put this in your template: <script>const myVar = '{{ my_var }}';</script>. This doesn't work if you put it in a separate javascript file, because that isn't rendered.
- Continued Altered Carbon. While a great show, I still don't like this genre. It's unsatisfying as a viewer to not be able to trust any scene you're presented. It's a deus ex machina on steroids, favoring the show instead of the audience. It's a complexity which hurts the viewer experience rather than adding depth and substance to its plot. Climaxes and resolutions in such worlds aren't clever; they're easy.
- Template variables are cast into objects (not just strings). This can cause complications between python datatypes and javascript datatypes. Strings are easy. For lists, use the tojson filter {{var|tojson}}. For others, look it up.
- Multiple elements within the same DOM cannot have the same ID. Use classes to group different elements together, if desired (even if there's no associated styling).
- SC
- Worked on commit_picks, parsing the data from the request all the way through writing it to the db.
- Updated the descriptions on some existing tickets based off new realizations.
- Serverside and clientside verification of all picks.
- Emails the user upon pick submission (optionally). Checks and balances, baby. At least until people trust the interface.
- Changed the table to use classes to group matchup rows and identify the specific columns. Previously, I was using js to find the header and do string matching, or explicitly stating column index. Ridiculous.
- Handled the submit button logic. Disappears during the ajax call, and reappears if any changes to the picks are made. The user is notified if the picks were committed successfully.
Wednesday, March 13, 2019
- Blogger's editor uses <br> and <div> tags instead of <p>, hence all the inconsistent formatting in these posts. Sometimes it will inject a newlines at the beginning of a post, sometimes at the end, and sometimes both. It changes all the time, even for a single post that hasn't been edited. Oh well, I can live with this. I've been happy with the other functionality.
- FW, MD.
- Registered for the Oct 5 Sonoma tough mudder with Eric!
- Even if the piece of info is as simple as a single variable, a (hidden) form is still the canonical way to send data to the server. You can do it through ajax, alternatively.
- If you are using ajax, and you're using flask-wtf csrf protection, you have to send the token (or exempt that route). Incredible help from Martijn Pieters: https://stackoverflow.com/a/22929593.
- Started watching altered carbon in the background.
- Supercontest
- Lots of work on the picks. Played with different methods of sending the pick array to the service. Ultimately went with ajax.
- Added the csrf token as metadata to the clientside.
Tuesday, March 12, 2019
- Back from Havasu.
- Smoked a 10lb pork butt.
- Verified mileage on all three motorcycles for AAA.
- Returned Amazon package at Kohl's.
- Booked Seattle trip with the companion pass for Allie's bday.
- HN
- Much better CLI prompts in Python: https://github.com/Mckinsey666/bullet
- Crunchbase is a cool site that shows lots of internal information about a company. For example, all of SpaceX' funding rounds: https://www.crunchbase.com/organization/space-exploration-technologies/funding_rounds/funding_rounds_list#section-funding-rounds.
Thursday, March 7, 2019
- Stanford Aero/Astro is starting an alumni newsletter now. There are 17 people in the new aero/astro undergraduate program already! Per Enge died :(
- I wonder how hard it would be to write a program that parses Ryan’s footage to extract stats like catches and hits. Amazon’s Rekognition tool can do action detection and pathing and such. That would work.
- Blogger is wayyyy less resource-consuming in chrome than Drive.
- Caught up on new season of Brooklyn 99. Didn't know it started back up again!
- Talked to 24 yesterday (Luis). He said that I get a free month but you still have to pay for two months. I assumed that meant that the first month was free, then I could just cancel before paying anything. Sneakily, it meant that you pay for the first and last month, and the middle month is free. I should have pressed harder.
- Netflix has a heavy ML and experimentation platform. There's a large team working on data science and research. I read their tech blog post about it.
- To discover private or hidden APIs on a website, just use the network tab of chrome devtools. It will tell you the url routes, the http method, and the data. Then just use it for you own purposes in whatever custom way you want! requests.post(url, data). It can get more complicated with cookies and auth, but you can copy that as well because devtools shows it!
- A shortcut - just right click the request in devtools and copy it as cURL, or convert it to python, or whatever you want. Very easy export method to repeat requests.
- sudo apt install gnome-tweaks to get more settings. Removed trash icon, added battery percentage, and a few more small improvements.
Wednesday, March 6, 2019
- Read a few more Netflix tech blog posts. Encryption, streaming, etc.
- Read some from Brendan Gregg. Pantastic perf resource http://www.brendangregg.com/.
- Moved blog to blogger! Better search interface (for my own posts) than Medium. Domain hosted by blogspot (free).
- Perf records total system activity over a time, then outputs files of the data for something else to analyze. Flamescope is the netflix (open source) tool to visualize this output and determine what was going on. Like a heatmap, it highlights areas of high activity with colors.
- Periodic railing - could be garbage collection. Periodic idling - could be blocking on i/o, or one thread has a mutex.
- Strace for system calls/signals, ftrace for kernel function calls, perf for general cpu profiling.
Tuesday, March 5, 2019
- MySQL is from Oracle.
- Read an interesting non-binary article that didn’t just discuss gender identity as a new thing, but race identity and age identity. It’s a new perspective to consider trans-gender like trans-race and trans-age.
- Told Petty about modern db explorers (he was going to build an asp.net application from scratch to read the db lol). Showed him desktop and web versions, lite sqlitebrowser and sqliteonline.
- Rather than 31+ years, I’ve been alive for 374 months, or 1629 weeks, or 11405 days!
- Netflix Tech Blog posts.
- Canary analysis. Deploy new version of software to canary systems to detect bugs. Basically diverting real traffic to a small canary cluster on the side of the production cluster, then comparing metrics and performance.
- Spinnaker is the continuous delivery stack. They developed it at Netflix, but now it’s used at Google, MS, Amazon, Oracle, many places.
- Kayenta is the platform that pulls both together. It makes canary deployments autonomous so users don’t have to do these tests manually.
- Chaosmonkey is public.
- Public repos!
Monday, March 4, 2019
- “Disable Cache” is on the network tab of Chrome’s devtools. It only disables the cache while the devtools are open.
- The em unit is usually the default browser font size.
- $() is jquery. Much better than the old getElement* syntax.
- Commas in css selectors means apply the rule to all. Spaces means hierarchy - only applies the rules to the latter elements that are children of the former elements.
- Css classes aren’t applied in the order you put them on the element, they’re applied in the order they’re defined in the source css document!
- Removing an element from an array by value is way more complex than it needs to be.
- Jinja injects newlines stupidly!
- Remember, var is function-scoped. The new ones for es6, const and let, are block scoped. Const is constant. Let can be reassigned (different than mutable).
- Supercontest
- I’m running Bootstrap 3.3.7.
- Fixed the rounded corners in the table.
- Smartly added the click functionality, restricted to the team columns, integrated with a submit button.
- Submit button now smartly appears and disappears based on picks.
- Used notify.js instead of flash.js to use the less-imposing messages rather than alerts.
- Warnings appear on >5 picks and opposing-team picks now.
Sunday, March 3, 2019
- Supercontest
- The local db was busted so I had to scp it from the prod server and upgrade to the schema with the user table again. Registered my one account to tinker. Copied the db to expedite this if it happens again.
- Moved the flask_user_layout template to my repo (instead of in the venv flask-user installation) to customize the year and company name, while making everything more consistent. My supercontest layout.html now uses the css classes header-div and clearfix and with-margins and stuff.
- Remember, if PyCharm is connected to the local db, the app won’t be able to write to it.
Saturday, March 2, 2019
Friday, March 1, 2019
- Checked with Rebecca Balayan, my SpaceX HR coordinator, if I had any outstanding mission patches.
- Opened the exercise request for my shares. I have 3 days to make the check payment. I’ll send it with Allie or Briley on Monday. Called home and asked for the 30k transfer. BOA has limitations so we may have to wire.
- There was an $86 charge from 24hr fitness on feb 8. I’ll dispute this back in Hermosa next chance I get.
- Instead of just instantiating a flask app, having a function return the app is called the “application factory pattern” for future reference.
- Docker-compose is often used to orchestrate two containers: one for the service and one for the database.
- Changed my default editor with sudo update-alternatives --config editor.
- When rebasing, the bottom section of the <<<< ==== >>>> is the newer commit. It will show the hash. The top says head. It’s the older one.
- When rebasing, “us” is the branch you’re rebasing on (master, usually). “Theirs” is the current branch.
- HN:
- Lol https://thisstartupdoesnotexist.com.
- JIT-compiled is just-in-time. It compiles as needed. Julia is an example of this.
- Cleave is the JS lib which formats the user’s typing on an input like a phone number field. https://nosir.github.io/cleave.js.
- Sync subtitles with video: https://github.com/smacke/subsync.
- Supercontest
- Deleted branches #2 and #7, they were folded into #3. Rebased #3 and merged. There was some nasty conflicting history. Closed the ticket!
- Started ticket #4, picks.
- Created a stripe account to process payments (https://dashboard.stripe.com/test/dashboard). Created a ticket to finish this work later (https://github.com/brianmahlstedt/supercontest/issues/29).
Thursday, February 28, 2019
- Talked to the Best Buddies coordinators. They do monthly social hours (last thursday of every month at Q’s billiards in West LA. I’m going to go to the next one.
- Night Crew Labs’ new website! https://www.nightcrewlabs.com/.
- PW
- Sklearn has a ton of cool machine learning capability. You can even do data analysis on text, finding correlations and such.
- WSGI. Request handling and routing is different than application logic.
- Squarespace is a very popular platform for building websites.
- Ag will automatically skip folders like venv, so make sure to take that into account when searching for stuff that might be in site-packages.
- Supercontest
- Reinstantiated a lot of the flask-user routes. Played with them a bit. Tried to customize most of it by integrating the default forms and such into the navbar of my supercontest page. Ultimately decided against the customization. I can do it later, and right now I want to focus on function rather than style.
- The navbar at the top is now in an ok place, with home, user, and signout.
- Played with PyCharm a bit more.
- SCP is great to move files between. It goes source -> destination. For example, to move the production db to your local dev machine, scp -i mykey southbaysupercontest.com:~/code/supercontest/data/supercontest.db ./test.db.
- python manage.py db migrate -m “description of change” after you modify the models. Then go to an existing db and python manage.py db upgrade.
- Pycharm allows you to enter an SQL console on the database in question, which is nice.
- If you have an existing database with a version that isn’t tracked in the current history (the migrations folder), then internet might tell you to init/migrate/upgrade. This is fine, but it’s a shotgun approach. It’s a wipeover. If you know what version the old database is close to, you can just go into the db and overwrite the value in the alembic_version table, then upgrade as usual.
- Copied the database over from the production server and upgraded it to the most recent model, which accommodates users and logins.
- When pycharm is connected to the db, it locks it, so the app can’t write to it. You can disconnect in pycharm with a right-click.
- The prod database now has a user! Me. I deleted the example user.
- Aligned the th.
Wednesday, February 27, 2019
- Played with Pycharm a little bit. Told it which venv/python to use for this project.
- Supercontest
- Added instantpage (https://instant.page/) to the main layout. Should speed page load with prefetch. Seems safe.
- Finished with both seasons of Ozark now.
- Got lunch with jcriss. Checked out the Redondo Beach Bay Club for the first time.
- Started reading the book The Four Agreements. I’ve heard good things. Overall, not very impressed. Seemed straightforward.
- Be impeccable with your word. No gossip. No lies. Pretend you are speaking on live tv at all times.
- Don’t take anything personally. Other people assume something about you? Ignore it, be yourself, and prove them wrong! This applies to compliments as well as insults.
- Don’t make assumptions.
- Always do your best. Take action to enjoy the action, not to expect the reward. Sometimes working twice as many hours won’t mean you get there any faster.
- Be a child! Be a dog! Explore the world with curiosity and delight.
Monday, February 25, 2019
- Submitted an application to volunteer with Best Buddies! Honestly teared up a little bit while reading about it.
- Bought and brined the lamb for tomorrow’s cook. First time smoking lamb! The costco business center has a pretty incredible selection of cuts.
- Macros: I’d like to aim for something like 100/200/80 for grams of Carb/Protein/Fat, which is a calorie distribution of 20/40/40%. This totals about 2000kCal/day.
Saturday, February 23, 2019
- Went and saw a live taping of America’s Funniest Home Videos (AFV)! Overall, was a lot of fun. Carlton is the host now. The audience is pretty involved. Comparing this production to my SpaceX webcast production was very interesting. They’re very similar, and I’m thankful to have the experience. Live is very different. Carlton could retake anything that had a mistake.
- DMV
- Registered the Ninja as PNO (planned non-operation). It’s no longer insured, as well. $22.
- Provided proof to the DMV that the BMW is insured. If they detect a vehicle is registered, but not insured, they will suspend the registration. I went online to provide this proof after I received the DMV letter, but AAA had already provided proof of insurance to the DMV, so the website informed me and all was good.
Subscribe to:
Posts (Atom)