mostly software engineering

Saturday, October 12, 2019

Supercontest.

Updated version to major release (not semantically for functional change, just needed to rollover). 2.0 baby.
Removed the modal edit/create. It goes back to redirect. But I also confirmed that editing does NOT work.
Getting cookie header errors for cloudflare, plotly, and discord:

A cookie associated with a cross-site resource at <x> was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.
I'll handle this later.

Chat clients.

Created a discord server and channel for supercontest.
Discord offers a native embed for a read-only widget. To allow users to write messages, you need a wrapper around the embed. Titan and widgetbot are two very common options. I went with the latter, because it's a crate instead of an iframe, and the docs/experience were much better.
Was broken at first due to ddos on most of their shards, but an admin gave me an alternative in the meantime.
Closed https://gitlab.com/bmahlstedt-group/supercontest/issues/22.

Removed the create/edit/delete from flask-admin. It didn't work, and this keeps other admin from severely affecting the db. Petty can still add late picks, see the scheduler, and view the data. That's pretty much it.
Deleted the 2018 paid league. It was identical to the 2018 free league because I didn't track that data back then. I didn't want people to see the 2018 paid league having more people than the 2019 paid league and thought it decreased, when in reality it's just that the app kept everyone from 2018.

As far as all players, we had 42 people submit at least 1 pick in 2018 and 45 in 2019, so there was some growth.

Added a few clearbit logos and listed out all the flask extensions on the readme. Looks good.
Added a pypi version badge to the readme as well.
Changed the #general text channel to #supercontest. Deleted the voice channel.
Created a FontAwesome5 account and added it the templates, just one script in <head> then <span style="color: COLOR"><i class="fas fa-ICON"></i></span>
Added is_paid_user to the g object, for the main and season blueprints, and added a corresponding query to get this info.
Stripe. The whole cycle.

Navbar indicator for already-paid vs want-to-upgrade?
Points to route that creates the stripe session and line items, then renders a template that simply redirects to stripe checkout.
Tested with test secret key, test public key, and test card information.

Changed bootswatch from united (orange) to yeti (black).
The scheduler worked for the unfinished picker email.
Sidenote - all admin have shared access to the supercontest gmail outbox, as an audit log for late pick submissions.

Finished season 1 of succession. Gonna try to get through as much of season 2 as I can. The season 2 finale is this sunday.
Wow. Employee fires back at employer? https://deploys.io/docs/disweb/.
Heisenbug - a defect that you can't find once you start looking specifically for it.
Vorfuhreffekt - when you try to present something and the opposite happens (like a software demo).
I agree with most of this, daily standups are more distracting than helpful: https://codethrasher.com/post/2019-10-06-the-pointlessness-of-daily-standups.
Full spotify in your terminal! https://github.com/Rigellute/spotify-tui.
The south park episode that talk about how ridiculous chinese censorship is got banned in china: https://www.vice.com/en_us/article/ywaq3g/after-a-south-park-censorship-episode-china-deleted-the-entire-show-from-the-web.
Able is a new alternative to medium, with a focus specifically for software. I like it. This would be a great alternative for my blog. I created an account, and then stopped once I saw that they didn't have a "search my blog posts" feature. So crucial, so obvious, but no one has it at first. https://able.bio/.
Prunes are dried plums. Dates are actually the direct fruit of a date palm tree.
Cool murder mystery in a sqlite table: http://mystery.knightlab.com/.

Get 2 witnesses from the crime_scene_report, 14887 and 16371.
Join all tables through interview to get the killer was at the gym on jan 9, male, membership id starts with 48Z, gold membership, and DL plates included H42W.
Search git_fit_now_member and check to isolate the times from 20180109 1600-1700. This leaves 48Z7A and 48Z55, which is 67318 (Jeremy Bowers) and 28819 (Joe Germuska).
Join to income to see that only jeremy bowers has a valid ssn. This is killer.
Check his interview to find info about the person he was hired by. Query that to find that it was Miranda Priestly, id 99716. Her annual income is 310k. She has no interview. Insert her into the solution table and it tells you you're right, break out the champagne.

Confirmed that my 24 membership is billed to my amex. When citi arrives, I'll have to swap over.
Remember, the ducati is about 150hp (peak) 75lb-ft (peak) 400lbs (wet). As far as standard power to weight ratios, anytime you have over 1hp/10lbs, you're in a really really fast car. The ducati is almost 4 horsepower per 10 pounds.
Guy broke the record and finished a marathon in just under 2 hours. Crazy. That's a 4:35 minute mile pace.
ERR BLOCKED BY CLIENT is usually because you have an adblocker running, not because the service is broken in some regard. Some good (non-ad) content looks like advertisement material, like icons and such.
CircleCI is a competitor to GitLab. Used by fb, spotify, gopro...
Sometimes you need to test webhooks on a dev server, like my situation with stripe/supercontest right now. Services exist to forward webhooks to localhost, exactly for this.

Tried ultrahook first, but they only have a ruby gem to run their server.
Used ngrok next. It's awesome. Easy to install, easy to set up, easy to inspect, easy to use. http://localhost:4040 is the interface for inspecting ngrok itself.
Stripe CLI also comes with a tool for this.
Oh wow, I take it back. http://serveo.net/ is the shit. You literally just run one command, no signup or anything, and you can pick the url, and it's static.

ssh -R bmahlstedt:80:localhost:5000 serveo.net

Friday, October 11, 2019

Chat.

tawk.to is the chat app for live chat (talk to users on your site, like "need help with x?"). It's free.
minnit.chat is the chat app for group chat (users talking to each other). It's also free, for max 40 simultaneous users. https://minnit.chat/. Created a room, played around with it - I liked it a lot.
Tried chatcamp.io as well. Their website was pretty bad. Broken register link, bad UX on the dashboard, poor documentation.
The problem with these group chat embeds: they force user accounts on THEIR platform. I couldn't find a single app that allowed you to simply forward the user accounts that you already have on your site. It's honestly as simple as a single unique key; even email would work just fine. You just need a string to display in the chatroom. I understand the design though. They want user traffic on their own platform, they don't just want you to use them for storage space and chat functionality.
Closed the ticket as won't-fix: https://gitlab.com/bmahlstedt-group/supercontest/issues/22.

I'm not sure if GitLab issues have a resolution field.
I love that GitLab correctly calls them Merge Requests instead of Pull Requests, but they went with the legacy "Issues" instead of "Tickets" which I'm less happy about.
Went through the tough mudder pics from our event and collected the ones with us. About 3 for each, at 3 separate obstacles. Uploading your photo and having them find your pics is easier than searching yourself.
Supercontest.

Closed the user league creation ticket. https://gitlab.com/bmahlstedt-group/supercontest/issues/67. Will reopen later if need be.
Closed the final "Long Term" milestone.
Last remaining ticket: stripe. I do wanna play with this one. https://gitlab.com/bmahlstedt-group/supercontest/issues/29.
Tried customizing the navbar for flask-admin. Copying the templates over isn't perfect because then you can't upgrade the pkg. In addition, they have a full html template, you can't extend your own header, so that's not ideal. This was the same with graphiql; I'm surprised. It's not hard, I guess they just don't expect you to embed such a full page as a subset of another page. I guess you could just use an iframe.

Neither graphql or flask-admin were designed for iframe embed. I'd have to subclass some of the internals to reroute some templates, and again, I'd be making it more of a pain to upgrade than anything else. It's fine as-is; getting my banner on top is not worth it.

Turned autocomplete="off" on the feedback form so the "petty's mom" surprise wouldn't be shown before you type, if you'd already done it at any point in history.
Went ahead and added the convenient late pick form in the admin view. It tells you the week and season, you pick the user and teams from dropdowns, then it commits and emails them. Nice.
Closed the ticket: https://gitlab.com/bmahlstedt-group/supercontest/issues/118.
Added Petty to admin.

Started playing with stripe. Submitted application to activate the account (business info basically).

They have Stripe Atlas, which is a pretty cool service that helps you form a C Corp, generate the docs, get a tax ID number, etc.
Used the wizard to start the "Checkout" tutorial. Accepting one time payments, from an https website, etc.

Got the TNF patriots pick correct. +17. Nobody picked the giants.

NE defense has the 2nd most points out of any offensive player (only McCaffrey higher) and they have the 10th most for avg/week

PPV.

As of Oct 18 (1 year ago), Mayweather Pacquiao is the best selling PPV of all time: 4.6m buys.
Mayweather McGregor is 2nd at 4.3m buys.
Khabib McGregor is 3rd at 2.5m buys. They drop off pretty quick.

Amazing UFC teaser commentary: https://www.youtube.com/watch?v=X7WywdwqK3o.
Steve kerr reaction to trump calling him a little boy about china dodging: https://streamable.com/8saxb.
Good summary by Buffett: https://www.youtube.com/watch?v=vl2aP8dlIn4. Human investing makes sense. People get pumped, and you get huge bull markets for short times, then they stay stagnant for years because they're looking in the rear view mirror, doubting it will continue so gold.

Now that the gigantic majority of trading is ML, this will change. Sure, you'll have explosions like crypto, but the general trend is more programmatic.

In basketball, true shooting percentage (TS%) weighs free throws lower and 3-pointers higher, so it's a more balanced metric for measuring your shooting efficiency than field goal percentage (FG%). You're penalized less for missing a 3 because it's harder, etc.

Steph curry got 97% TS yesterday: https://www.youtube.com/watch?v=qftr409jP-4.

"Vegetable Oils" are usually just blanket terms for trans fats. You're better off with a seed oil like flax or sesame oil, because you're getting the actual oil and fat from from the source.
Cool inside look at the nutrition program for the patriots: https://www.youtube.com/watch?v=9iDTAzD2hT4.

They get every meal provided, even in the offseason.
You can customize and special-request items, if you want.
They go through 70lbs of poultry, 25lbs red meat, 25lbs fish, and tons of vegetables every day.
Everything organic, grassfed, freerange. The usual.
They do a ton of smoothies.
The whole program is surprisingly equivalent to my daily meals.

Read up a little bit on bootstrap forms and validation.
You can have subsequent form inputs base their options on previous selections (e.g. season -> week). You have to put these conditions, and subsequent queries (post to server for new choices) in javascript. It's dynamic. There's no shortcut.

Thursday, October 10, 2019

Gitlab/sentry:

I don't think you can default an assignee for tickets or MRs in gitlab (yet).
Moved my project from my user namespace to a group. This is like an epic across projects, combining them together to see all tickets, etc. You can also see cycle analytics.

Updated my remote urls to bmahlstedt-group/supercontest.git (updated namespace). Did this on my local machine and sbsc prod.

Resolved the old AnonymousUserMixin sentry issue.
Added the supercontest repo to sentry, and integrated the app in gitlab, so all my source commits are available to sentry.

Blandish - to flatter someone manipulatively.
MLB postseason is now in the semifinals. Nationals and Cardinals in the NLCS, Yankees and Astros? in ALCS.
The carnivore diet is absent of vitamin C, vitamin E, and fiber.
flask-user utilizes flask-login, not flask-security.
Tony Bennett is currently 93 years old. Frank Sinatra died at age 82 in 1998.
Supercontest.

Fixed a few small bugs. Some wording.
Added sentry releases.
Rejiggered the picks_open logic to natively include is_current_week. Restructured the IF logic in the your-picks/all-picks routes to be extremely clear about the scenarios, and which info bubbles to present to the user based on timing (no lines, no picks, post-picks, picks-closed, etc).
Integrated flask-admin.

The main benefit is that you get ModelViews out of the box, allowing a web interface to create/edit/delete/search the database.
The secondary benefit is just organization. My dashboard, scheduler, and all the other custom admin views for the supercontest can be added to the general /admin navbar.
Added the model views. Protected with CRSF (flask_admin.forms.SecureForm) and restricted to an authenticated user with the admin role.

I now have 8 tables in the app.
There was an apscheduler os write error that sentry reported: https://sentry.io/organizations/bmahlstedt-org/issues/1266918507/?project=1773879. Not sure.

Remembering 15 years ago to college applications, since I'm starting to do the same with job apps; I'm much more of a sniper than a machine gunner.
I've used ~100m of 2000 (5%) of my shared runner pipeline quota.

Wednesday, October 9, 2019

Supercontest.

Added a global @app.before_request to print the email of the user for all requests, to make the logs more clear.
Made the matchups js a lot smarter about iterating through picks. Instead of going over all matchups rows and comparing to picks, it gives the ids to the td cells and then targets the specific row, just iterating over the picks instead of all matchups.
It now adds a 2px border to cells for complete games, to distinguish between what is finished and what isn't. This happens on the your-picks and all-picks views.
Fixed the emailer. The problem was the asyncronous job adding. APscheduler is a piece of shit, from all my experience with it so far. Jobs added before scheduler.start() will trigger properly, but having another view call add_job later has never successfully triggered that job. Now, it just adds the email reminder at app start (for prod app only). I left the scheduler admin view around, but put warnings that it doesn't really work due to apscheduler.
I've been more conscious about pushing, now that I have CI set up. I batch a lot more local commits now. This will save the 2k min limit on the shared runners.
In setting up CI/CD on the droplet yesterday, I had accidentally removed the public key in /home/bmahlstedt/.ssh/authorized_keys (the one that corresponds to my private .ssh/digitalocean). I added it back (/root/.ssh still had it), and I could ssh again.
In moving source checkouts over from github to gitlab, I lost all the backups (they aren't tracked in vcs). This is fine, I have a new one with all the current data.
Changed the matchups template to use picks_open as well.
Added cards to the messages above each view's table. Styled better.
Added the roles table, create admin, added myself. Updated all views to use this properly. https://gitlab.com/bmahlstedt/supercontest/issues/115.
First MR, ticket closure, master kickoff, deploy; all that ci/cd went well in gitlab.
Wednesday deployment, changed banner.
Quick bugfix with picks: https://gitlab.com/bmahlstedt/supercontest/issues/117.

Sentry:

List of importable integrations: https://github.com/getsentry/sentry-python/tree/master/sentry_sdk/integrations.
Added sqlalchemy.
You can't add the gitlab integration for sentry unless your project is in a group (not owned by single user) - https://github.com/getsentry/sentry-plugins/issues/458.
Changed sentry to only run in prod mode, so the local dev instance doesn't eat up my sentry monthly event limit.
The uncaught "AnonymousUserMixin object has no attribute id" errors were from queries.is_user_in_league in the url_value_preprocessor. I protected that with if current_user.is_authenticated, so we shouldn't see those anymore.

Protected all other current_user calls too (FMD's get_current_user_email, etc).
Note for future apps: if you're ever calling current_user outside of a view or other function where the user is guaranteed to be logged in (protected by @login_required or such), you should condition the current_user object on is_authenticated.

Left the email notifications for every exception ON. I don't anticipate spam, now that the AnonymousUserMixin error is fixed.

Flask debug toolbar changed its default to disable the profiler. I re-enabled it.
Lol not the biggest jon jones fan but completely agree with this: https://www.reddit.com/r/MMA/comments/df49ql/jon_jones_to_izzy_im_not_out_here_searching_for/. Izzy is annoying.
Created an radnet account to access my MRI results.

https://myradiologyconnectportal.com/Exam.
Bad: "There are mild degenerative changes of the coronoid process."
Good: "There are no areas of abnormal signal involving the trochlea or capitellum. There are no areas of abnormal signal involving the radial head. The radial collateral ligament and attachment of the common extensor tendon and ulnar collateral ligament and attachment of the common flexor tendons are normal. There is no triceps or biceps tendon tear. There are no soft tissue masses. No ulnar nerve lesion is identified."

You can make waiver claims on tuesday night for ff. I had thought that it was wednesday all season so far.
Ortho appointment in Van Nuys. Basically just said it's ok, go to PT. 10 appointments concluded with 0 treatment.
Picked up DL from Brews Brothers after.
Ate lunch at Dave's Hot Chicken in NoHo. It was amazing. First time I've been impressed with outside food in a long time. 20min wait around noon.
Loving Terry Reid's music. Always liked seed of memory, but had never heard faith to arise or any of the others. Unfortunately, that whole album isn't on spotify. He sounds a lot like Robert Plant, and the music is chill.
pylint no-else-return has always been a dumb rule. Flow is so much more clear when it's explicit about the else.
I've said it many times before, but unfamiliarity is the root of most discrimination. Racism would be nearly extinct if the whole world spoke the same language. Then you could tackle the minority remainder, the heart of the problem; anger.
In gitlab, you don't need to put a pound sign in the branch name! This makes command-line-git a lot easier. You still put it in the commit message though.

Tuesday, October 8, 2019

My 15(?) year old wix website has finally been cancelled (which is probably for the better).
"And the day came when the risk to remain tight in a bud was more painful than the risk it took to blossom." - Anais Nin
Finished Once Upon a Time in Hollywood.
Smoked the pastrami beef ribs that had cured for 6 days. Amazing flavor.

Next set of beef ribs, I want to try wrapping earlier, closer to the time you would for brisket. I'm curious how much it changes tenderness vs bark.

Paid off the remaining balance on the sequoia hospital bill from years ago; was like $220.
Cancelled the amex.

They tried to give two incentives, one for miles (20k) and one for cashback ($175), both requiring that you spend 3k in 3 months.
The companion pass is forfeited (I had 1), and the annual fee is not prorated (there were 4 months left in that this billed year).
Removed my amex from google play. Removed the amex browser bookmark. Mint automatically made it inactive, I didn't have to do anything!

I would like the Vitamix A3500 - it's so much better than my ninja, and I use it at least once daily. It's $525. I'll probably wait until after I move.
The Strassburg sock is definitely helpful for plantar faciitis. WFH makes it a lot easier to get hours in during the day, since I find it annoying to fall asleep wearing.
Put the 2020 reg on the Ducati. It's yellow, so it looks better than the blue 2019 sticker.
Watch Rob Zombie's new 3 From Hell, the third installment in the series after House of 1000 Corpses -> The Devil's Rejects.
The carolina reaper single chip is sold out online, but I just discovered that the riteaid on aviation/artesia has these 3 flavors:

I like the haunted ghost pepper the best, then salsa verde, then chile limon.

Kubernetes/GCP.

Good article on kubernetes: https://www.digitalocean.com/community/tutorials/an-introduction-to-kubernetes. Very useful for orchestrating many containers across many machines together. Metrics, storage options, scalability, ML resources, much more through the full GCP suite.
Created a project (supercontest) on GCP and removed the default API Project and My First Project.
Enabled the Google Compute Engine API for this project.
n1-standard-1 machine, just 1 vCPU and 3.75G mem (the smallest standard machine).
Billing is by resource. It's like 3 cents per cpu hour and 0.5 cents per gb mem hour, so the n1-standard-1 is about 5 cents per hour, or about a dollar a day, or about $30/mo. My digital ocean droplet is $5/mo.
With the $300 promotion on signup, looks like I can test this for free pretty easily.
Ended up staying with digital ocean for now. I don't need any of the other GCP products (compute, storage, ml, etc). The one attractive feature is scalability, but I can upgrade with DO as well.

GPG key = GNU Privacy Guard.
Sentry.

Created account, added the flask integration to supercontest.
For the free plan, you can record 5,000 errors per month and it keeps history 1 month back. That's 160 errors a day.
Linked my gitlab project to the sentry api, created auth token.

GitLab.

If your project is public, and you have gitlab ultimate, then you get a security dashboard and dependency list. Pretty cool, but I obviously can't take advantage of this.
In general, I've been very pleased with the experience on this platform. Moreso than the atlassian stack, jenkins, etc.
Added the deploy stage to my yml. Added the variable SSH_PRIVATE_KEY (for the droplet) to gitlab. Added it as a known host directly through ssh-keyscan.

Gitlab runs the deployment as root, and the deployment has ansible pull new source on southbaysupercontest.com, so I had to make sure the machine had the proper ssh keys under the root account (not just mine). sudo cp -r ~/.ssh/ /root

Ansible doesn't maintain docker images officially anymore, but some user groups keep them around (albeit not very current). Also...you have to specifically install ansible after you pull the image lol.
Did not integrate prometheus; there's already a metrics agent that digitalocean provides to its droplets. I put a link to the external DO dashboard in gitlab tho.
Updated the readme to indicate where metrics were, as well as SCM, CI/CD, etc. The GitLab and DigitalOcean logos (clearbit) look good.
Modified my view to be fluid.
The minutes you've used in CI/CD are at https://gitlab.com/profile/pipeline_quota.
Integrated with sentry so that error tracking was in gitlab as well.
I don't need to use the container registry associated with the project, although this feature is great. My workflow is not build -> test -> pushImage -> deploy, so a record of all the image artifacts in a registry is less important. I test the source and then build the image right at deploy time, which is more than satisfactory until I start scaling the test/review app portion of things.

Sometimes dpkg will prompt for config changes (like apt does for install y/n). Use -o DPkg::Options::="--force-confnew"
Changed my mem usage alert on the droplet from 90 to 95%.

Monday, October 7, 2019

Gitlab.

Migrated all projects from github to gitlab.

Added the public ssh keys from both bmahlstedt-xps13 and southbaysupercontest.com.
Delete the checked out repos, then recloned from the gitlab url (you could also just set the remote url).
Privatized and archived all my projects on github, with a pointer to gitlab.

Cool features:

Activity charts and filetypes: https://gitlab.com/bmahlstedt/supercontest/-/graphs/master/charts.
All contributor information, and when: https://gitlab.com/bmahlstedt/supercontest/-/graphs/master.
Great CI/CD, security/compliance, operations, package support, more.
Commit heatmap: https://gitlab.com/bmahlstedt.

Got initial CI set up to run test-python on commits. Using shared runners for now, 2000 min/month should be enough. The wait times are zero. If needed, I'll register my own machine as the runner (since I'm the only developer, and the machine is guaranteed up).

Using custom stages, rather than autodevops. That's a cool feature though.

Added build and coverage badges to the readme.

Google Cloud Platform.

Kubernetes is obviously the google production container orchestration platform. Gitlab has lots of integrations with it, as well as Google Cloud Platform in general. I should try it out.
GKE = Google Kubernetes Engine.
Started a trial with GCP.

Removed pay-with-points from my amazon card. You're losing the cashback if you pay with points. Just always use the card, and withdraw all the cashback from the chase account whenever you want.
Homemade protein bars for my mom yesterday. Pecans, oats, prunes, cinnamon, protein.
Won both fantasy games. Went from 0-4 in yahoo, last place, to highest scorer of the week. 2.5 in sbsc.
Placed fresh order.
Supercontest.

Put in the niners pick for jcriss and econstan.
Changed FMD to monitoring level 1 for the user and scheduler blueprints, while leaving my main and season blueprints at 3.

Finished the algorithms and data structures book, added all notes to google drive.
Ordered a small bedroom trash can, since amazon lost the other delivery (got refund).
I would like to do a hanging display for all the tough mudder headbands I have. The most basic is just wood blocks, maybe 1x8" with 1/2" thickness, then hook screws connecting them on both sides.
There's a ufc event every Saturday for the next 6 Saturdays, and they're all good.

Sunday, October 6, 2019

Tough mudder yesterday was great. It was on the easier side (only 8 miles), but I saw a few new obstacles and got the pink headband. Awesome to do it with Eric. Plantar fasciitis did surprisingly ok.
Fogo de chao. Prime rib and filet were good, as always. Beef rib, sirloin, new york, pork rib, picanha, chicken - everything else just can't compare to the smoker.
Cal lost to the ducks, warriors lost to the lakers (pregame).
Watched the comain and main for ufc 243. Dan hooker is awesome. Adesanya looked fantastic but is annoying as hell. Too cocky and had one of the dumbest walkouts I've ever seen. Tried to make it a choreogrpahed performance like WWE and boxing. Gtsoh.
Supercontest

Entering the python shell is not resource-cheap. It creates an entirely new instance of the app. Not with a server, but will all the other frills; apscheduler, etc.
Submitted late picks for Cam, Wes, Raj, Jesse Thompson, Ben Yezer, and Phil Twist.

Credit card research.

BoA preferred rewards offers 5.25/3.5/1.75, which is pretty damn good but you have to have 100k+ in your combined BoA/Merrill accounts. Otherwise it's 3/2/1. If you have that much sitting in your account, then you can gain a lot more by moving it somewhere with higher growth rather than reaping higher cashback.
While looking at NEW cards, I saw that there's a better version of my old card. This is a ridiculous mistake. I've had an Amazon Rewards card since July 2013, which earns 3% back on Amazon. If you have Prime, which I do, you can upgrade to the Amazon Prime Rewards card, which gets 5% back. It did not automatically upgrade me, and I never noticed. I exported an order report, and I have spent $30,000 since July 2013 on Amazon. I missed $600 free dollars by not clicking the button. Chase will ship a new card automatically, even if the upgrade was done on amazon.com (I called Chase to confirm).
Made my last payment on AmEx Platinum Delta Skymiles, will cancel in the next couple days. I no longer do business with Delta. The miles won't expire as long as I maintain my Delta account; the amex card is not necessary. I can finish off the remaining ~135k miles whenever I want, as well as the buddy pass this year.
Looking for purely cash back. Gas is easy with the motorcycle. Not a big traveler, don't need air/hotel. Already a bomb chef, don't need restaurant rewards.
Chase ultimate rewards program is good for certain use cases. You can regard all "points" as basically 1% cashback, with some variation. In reality, a point can be anywhere from 0.8 to 1.5 cents, and can change based on what you're redeeming them for as well. The most common use case is for travel, where you can get 1.25-1.5x the value of your points. You can also share points between cards, which is useful for people with business cards - they can use the points earned on work trips toward their personal rewards program. It's only raw chase cards - cobranded cards (like my amazon) don't share. Because of this, and the de-emphasis on travel, I don't need a card in the chase ultimate program in particular.
Cashback-focal cards with no annual fee and no rotating schedule:

Citi double cash. 2%.
HSBC cash rewards mastercard. 1.5%. $150 bonus once if you spend 10k in first year.
Paypal cashback mastercard. 2%.
Capital one quicksilver cash rewards. 1.5%. $150 bonus once if you spend 0.5k in first 3 months.
Chase freedom unlimited. 1.5%. $300 bonus if you spend 20k in first year.
(Chase freedom doesn't count because they rotate categories, which I don't need).
Amex cash magnet. 1.5%. $150 bonus once if you spend 1k in the first 3 months.

Decided to go with citi double cash. Highest basic cashback. I already have visa through my amazon prime rewards card, and this one is mastercard. Applied and accepted.

Common, boring mistakes: https://medium.com/signal-v-noise/the-4-questions-you-should-stop-asking-during-your-one-on-one-meetings-ed7431da11aa.
Cool NFL data of score combo possibilities: https://nflscorigami.com/.

Friday, October 4, 2019

Redis can read a json 16x faster than postgres.
"Vice signaling" (as opposed to virtue signaling) is a cool trait. Admitting one's faults, weaknesses, etc. I like that in people.
Petty finally sent me the list of people who had paid so far. Added them to the prod table, lb looks much better now.

enter-python-prod
paid_league_2019 = db.session.query(League).filter(League.season_id == Season.id, Season.season == 2019, League.name == 'Paid').one()
user = db.session.query(User).filter(User.email.like('%petty%')).one()
paid_league_2019.users.append(user)
Repeat 3-4 for any other users
db.session.commit()

33 so far. More are planning to pay (Art, Wes, Harner, ...).

Thursday, October 3, 2019

Supercontest.

Created and finished a small handful of improvements: https://github.com/brianmahlstedt/supercontest/issues/110.
Changed that stupid logic where every new endpoint would have to be excepted in nav_active. Now it actively highlights profile if it sees edit_user_profile in the route. If there are others in the Profile bucket, I'll add them as they come.
Fixed the FMD, adding grouping by user email. The actual problem was solved by the docker system prune and docker volume rm (for the fmd only).
Added an awesome admin navbar (custom).
Redesigned the whole schedule interface. No jobs are automatically started anymore, and none are started at application started. Wrote a nice interface in the admin panel that allows you to kick off each job individually, and view the currently running jobs.

Split it so that the lines job does not trigger the scores job. You must add both (and then they auto-trigger to actually start, of course).
This makes it nice because you can individually tell the scores job to look in the db at already-existing lines and add future jobs for them. Otherwise, if the app crashed after the lines were ingested on wednesday, you'd have to go back in and add the score jobs again manually.

Continually unimpressed by how fickle apscheduler is. Having fought more against it, I decided it was too unreliable to use in production. It was committing duplicate lines, not running at other times, and much more. It can't even report its own status accurately. I'm going to leave it in place for the harmless, non-self-modifying jobs (email reminder). I'm not going to use it for the repeat-until-post westgate line fetch, or introspect-live-scoring.

MemoryJobStore is fine if you don't need to persist jobs across something like an application crash. If so, the SQLAlchemyJobStore would be easy to change to. You can use sqlite or postgres.
The usual docker system prune -af, on both dev and prod envs. Wiped the volume for FMD, not the regular sbsc db. The usual git remote prune origin as well.
You have to pass the csrf token in form submission (if the action is going somewhere that has csrf protection, like my site). It's pretty easy: https://flask-wtf.readthedocs.io/en/stable/csrf.html#html-forms.

Wednesday, October 2, 2019

Played with the new metrics agent on the droplet. It's properly reporting data now.

Created an email alert for mem > 90% for 5 min.
I'll grab this again over about a week, to see mem usage increase in O(days), as well as sunday action.
CPU usage, disk i/o, process load - these are under control always, even on sundays. They only rail during a deployment..

Ducati hit 6000 miles today.
SF has a ton of food transportation startups.
uWSGI needs to re-enable the GIL with --enabled-threads in order to work properly with APScheduler.
Apscheduler doesn't expose if a job is running. Absolutely ridiculous.
Chiro
Jinja blocks {% block %} cannot be conditional, you must put the {% if %} within them, not around them.
Started the brine for pastrami beef ribs. Basically just make a usual rub, but with about a tablespoon of pink curing salt. Bring half gallon of water to boil with the rub mixed in, the add the other half gallon of water to cool and then submerge the short ribs for a week.
Supercontest.

Added ADMIN_EMAILS to supercontest.util.email. Made each schedule notify me of start/stop as appropriate. Confirmed the following:

Email reminder runs once, and emails me the summary as admin.
Line fetcher prints every minute.
Live scoring verifications, in order:

Comment out all of commit_scores until the live definition, then modify as necessary.
Add to the bottom of get_app() before the scheduler.start() call:

import datetime
from supercontest.schedule import add_live_scoring_jobs
datetimes = [datetime.datetime.utcnow() + datetime.timedelta(minutes=1)]
add_live_scoring_jobs(datetimes)

Visit the /scheduler endpoint during all of these tests.
Tests:

Scores are fetched every minute if a line's datetime is reached.

Shim the datetime input to add_live_scoring_jobs() with one datetime and live = True.

Score fetching only runs one job when multiple games are active.

Shim the datetime input to add_live_scoring_jobs() with 2 datetimes at the same time. Keep the live status.

Score fetching is stopped when all statuses are F/FO.

Set live = False. Make sure it emails you.

Future schedule jobs remain after removal of the current.

Add a third datetime before the other two. Leave live = False. When it is removed, make sure the next two are still scheduled in the future.

Then undo the first two actions.

New lines end the fetch job, and an email is sent.
I'll have to verify that the score fetching works fully, with real data, tomorrow.

Fixed the cyclic imports by lazy-loading the wrapped functions inside the job targets.
Adding a new navbar for admin (dashboard and scheduler). These do not participate in the nav_active regime.
Updated banner, usual wednesday deploy.
Deployment takes about 4 minutes.
Was getting a weird KeyError: 5 in the limbo period between 5pm when the db week turnover and the new line commit happens. Could not reproduce in the dev env. I'll check next week again. Could have been some compound issue with the broken line fetcher (hopefully...bc that's fixed).
Added protection that if the lines are already in the db for the current week, remove the job and stop (don't commit scores, don't run another fetch again, don't email, etc).
Split up the templates into a navs/ folder. SO much better now.
Consolidated all the menus into one smart huge navbar. The template hierarchy is a lot cleaner too. All of the nav dropdowns are sequential, and then the final view extends them + fills out the main_content block.
Every view has league now. All of the conditional if statements to build url_for() calls are much simply now. It's just one branch at max, checking if week is needed.
Played with the responsive capabilities of navbars: collapse at certain breakpoints, splitting into different submenus, etc.

Remember that to delete a row with fk constraints, you must delete those rows that reference them (for example, you can't delete line rows until the score rows that reference them are deleted as well).
The jinja block you're filling doesn't have to be in the direct parent you're extending. It can be way up the lineage.
The href="#" on an <a> tag tells the browser to treat it like a real link. Without that, it won't show the click cursor (the hand) - instead it will show a regular pointer or typing cursor.
FDT needs a body tag, so add it if you want the toolbar for even basic endpoints that return pure html or raw strings.

Tuesday, October 1, 2019

I'm 0-8 in fantasy so far!
Appointments.

Got the MRI. Much better waiting room, they got me in right away. Photos took 45 minutes. I was lying in full superman position without moving, which became horrible with pins and needles toward the end.
Orthopedic = musculoskeletal. I had incorrectly thought it was only bones.
Made appointment for the followup in Van Nuys. Would have done Culver, but I needed to get my license back from NoHo anyway. You can't replace a DL through the AAA DMV services, and the soonest appointments for the Hawthorne and Torrance DMVs were December.

Supercontest.

I had to set MNF's status to F for some reason. It was stuck on 4, even though the NFL xml was returning F. All of my score updating improvements today will change that bit.
Last week was the worst league-wide performance since the beginning: 31.9%. Nobody got 4 or 5, 3 was the highest. There were four 0s.
Cleaned the main __init__ quite a bit (the internals of get_app()). All extension capability is well organized and properly imported now.
Split the determine_coverer/points logic into the wrapper (which expects rows) and the wrapped (takes raw values). This was useful because certain pieces of the app call these functions in those two separate ways.
Generalized get_matchups and get_picks to be more modular, rather than using specific wrappers for each distinct queries. It's going to be easy to manage, the args basically mirror the cols 1:1, then the filters just add whatever was requested.
Change it so points and coverer are now BACK in the db. They are written to by the apscheduler job, depending upon when games are live. They are read by everything (mostly in results), which simplifies a lot of duplicate logic.
Machine went from ~300 to ~600M in mem usage over 12 hours after restarting the app but not doing anything (like a deployment). Not good. There's clearly a leak somewhere.
Used a psql CASE WHEN THEN ELSE END statement for the migration, to populate old coverers and points. Just like an IF statement in a higher level language.
Added the proper apscheduler jobs for live game scoring, as well as line fetching.
Created the tickets for the next two (should be final) efforts:

Made the jobs not run in the off season.
Updated docs.

No thanks: https://gfycat.com/miniaturespanishkronosaurus-adventure-v-travel-nature-matterhorn.
What few colleges teach you about the fastest escalator to corporate success is the art of asskissing. Relationships (politics) mean more than individual contribution (skill) in most offices.
7'9" is the tallest anyone has ever been without any of the usual disorders that cause extreme height (gigantism, etc). https://en.wikipedia.org/wiki/List_of_tallest_people. He was just a statistical anomaly, the furthest end of the regular human height range.
Started the final 8 mile training run and my left foot nerve pain was too much to continue after ~3.5 miles.

Running was more painful in the knees/hips/back when landing on the heels, but I couldn't mid/toe strike because of the injury. Barefoot running really is so much better, but I'll have to be more gradual with training to strengthen my feet/ankles/calves.
I will definitely run the race in the full purple shoes, with tape over the right outer hole that had formed (will donate after race).
Re-downloaded strava to start logging the training again.

flask-apscheduler attaches the actual apscheduler base scheduler at scheduler._scheduler, which you can use for some stuff that the flask extension stupidly hides (like print_jobs()).

Monday, September 30, 2019

Abdulrashid Sadulaev is a 23yr old Russian wrestler. He is the current world champ. He's from Dagestan, just like Khabib.

Awesome pin of the defending champ snyder with some offshoot peterson roll all the way to fall? https://www.youtube.com/watch?v=Ryy19IY9fQo.

Supercontest. Worked on async score fetching and result calculation today.

Design decision: Score.coverer and Pick.points will update in realtime, as the scores change, rather than waiting until the game completes. This allows you to present mid-game date as "what if the games ended right now?" For views that don't want to bank games until they're done, like the leaderboard, you can filter with an additional check for FINISHED_STATUSES.
If the app is down for a game, you have to manually enter the information into the db. This is fine. Based on the xml scoresheet that I pull nfl scores from, the only option is live. In order to get past data, I'd have to hit another API.

sqlalchemy difference between default and server_default: default is just what python will insert for the value if you don't provide one, server_default is a statement that you can use to generate the value on the db side.
Chubbyemu videos are so interesting and informative.
The patriots, the chiefs, and the NINERS are the only teams still undefeated.
Ordered pink curing salt to make pastrami beef ribs with the next 2 racks. Curing salt is basically just table salt (sodium chloride) mixed with sodium nitrite. The pink doesn't matter, it's just dyed to make sure people don't use it as table salt (although pink Himalayan salt can be used as table salt, and it's obviously pink too). Regular sea salt can be used to cure/brine meat as well, obviously, it just can't go as long because the nitrites are what protect the meat from bacteria growth during the curing process.

Friday, September 27, 2019

Supercontest.

Added tons of stats for the history of the league. https://github.com/brianmahlstedt/supercontest/issues/23. Super interesting.
The submit button stays up after someone clicks it. That was the problem. Fixed it and removed a few more duplicates.

https://github.com/brianmahlstedt/supercontest/commit/4598563b71cec4525e6ac1edbeaeb8e654398692
You could technically still write a client that wraps the /pick api and blasts it directly over http. The uwsgi workers would all handle the requests in parallel, open separate connections to the db, and add all the duplicate picks. I'm not going to worry about this case right now.

cache.memoize on get_picks was causing problems if you made multiple pick submissions within the span of 10 seconds, because the client would be told about picks that might have been old, desyncing from the actual picks in the backend db. Fixed (by removing the cache for that query). Pick information should be live on request. Scores are different, because the user doesn't write to the scores table from the frontend.

Completely cleaned and reorganized my room. Cleared a bunch of space. Including closet, lights, everything.
DigitalOcean.

Did some research, you can upgrade any of disk/cpu/ram. It adds a decent amount to the monthly cost. Just going from 1GB memory to 2 makes it $10/mo instead of $5.
Allowed password login on the droplet for my user, so that if anything goes wrong remotely, I don't need my laptop and ssh key. I can just ssh from any machine.

sudo vim /etc/ssh/sshd_config
PasswordAuthentication = yes
sudo service ssh reload
sudo passwd bmahlstedt

Confimed that I have access. You can ssh in like usual or you can go to the DO website in the browser and access a console from there. Password is one of my old SpaceX ones.
Upgraded from the legacy metrics agent on my droplet to the current one.

tmux shortcut reminders.

To detach from your current session, just type your hotkey then d
To kill your current session, just type your hotkey then : then kill-session
If another client has another session in a different window size, just kick them off when you attach with tmux attach -d

Thursday, September 26, 2019

I care much less about the API utilization metrics that FMD provides. The most interesting is the API performance. Graph below. I wish they had multi-version api perf like they do for utilization. The results are mostly expected, but still an awesome feature. https://southbaysupercontest.com/dashboard/api_performance.

In checking back, you can see multi-version api perf for specific endpoints! Just not an overview of all routes for the application.

South Park is back!
FMD

Verified that it CANNOT do patchfix version resolution. Only major and minor.
A major version (3) was released yesterday, which appears to have broken the overview page. 3.0.3 was released a couple hours ago, looks like they've had a few bugfixes. Upgraded, and confirmed the overview is still broken.

Maybe this line of questioning will help the lesser perspectives understand the ignorant racism in the NBA owner concession:

Is it ok to compare someone to Saddam Hussein, when they have nothing in common with him except that they're Arab?
Is it ok to compare someone to Hitler, when they have nothing in common with him except that they're German?
Is it ok to compare someone to slaveowners, when they have nothing in common with them except that they're white?
It it ok to compare someone to <anyone bad or good>, when they have nothing in common to do with it except that they share the same race?
No. None of these comparisons are ok. They're offensive. Doesn't matter the historical subject/object, victim/aggressor, small/large, green/orange, old/young.

All daemon threads will be killed when the last non-daemon thread dies. The non-daemons are the main ones. The daemon threads are subservient - they will not hold the main app.
Supercontest.

Got a few "AttributeError: 'AnonymousUserMixin' object has no attribute 'id'" errors in the production app, with an IP from singapore. This just means they tried to access a route and weren't logged in. The flask_user.current_user object is AnonymousUserMixin when the user is not logged in, so they can't do anything. The error was caught pre-route in the url_value_preprocessor.
Right now, I have 94 compiled requirements (including testreqs) and 26 input requirements (not including testreqs).
Remember that while 99% of requests make it to the app and you can check the standard uwsgi/flask logs (make logs-prod), there's a small chance that nginx catches something. just run `docker logs -f nginx-proxy` to check.
Added text for harner's banner.
Tons more work on the redesign of blueprints, routes, url_prefix, url_defaults, url_value_preprocessor.

Used request.url_rule.endpoint and request.url_rule.argument to infer all of the values (in g) that the current view would have access to. Then I conditioned the url_for() navlink builds in the templates based on those.
It's all in a great place now. This was frustrating for a few days, but will be much easier to manage/change in the future without breaking things.
Closed https://github.com/brianmahlstedt/supercontest/issues/102.

Added apscheduler (advanced python scheduler) for the weekly email pick reminder.
Not sure how, but two people submitted 10 picks into the db. Both were pairs of the same 5. I deleted them manually, but I'd like to know how this happened.
Split up the utilities module into a separate util package, containing distinct modules for the specific type of functionality they provided. It was getting too big.
Originally wrote the email reminder with apscheduler, but there's a specific flask-apscheduler that made it a little easier. Don't need to manage the debug app duplication, atexit, or any of that peripheral wrapping.
Tested with my email only and a modified cron. Worked! The apscheduled job shows up in the regular flask logs too.
Closed the weekly reminder ticket: https://github.com/brianmahlstedt/supercontest/issues/97.

In debug mode, flask loads two instances of the application. If your app does something like schedule an external cron job, it will happen twice.
Added Google Keep back to phone and to browser bookmark. I was missing the ability to immediately open something and jot a note down, especially while I was out and about, away from my laptop/blog.
The `date` command in bash gives the local datetime with tzinfo.

The amazon rewards card gives you 3% back at amazon. There's also a "reload" feature on amazon, where you put money in your account (for use at amazon) and it gives you 2%, plus $10 bonus sometimes. I tried reloading WITH my rewards card today. It won't let you.
Electric toothbrush finally died, bought a new one. $50. Arrives tomorrow.
Went to ortho in Van Nuys, got an MRI referral. Total round trip: 2.5 hours. 1.5hr driving, 1hr in waiting hour, 5 min visit with doctor.

So far, the entire chain has been:

spacex doctor -> spacex pt -> general doctor -> bloodwork -> general doctor -> xray -> general doctor -> ortho doctor

8 appointments, 8 visits, 0 conclusions. The referral system in medicine is terrible.
There will be more. MRI -> ortho doctor -> whatever is next (hopefully not just PT).

Idea: there should be a global medical database. You have to fill out the same new patient forms at every clinic you go to, and 99% of it is the same.

Every office would have auth keys to a specific subset of the db: general information, surgical history, family history, ortho, respiratory, psychological, whatever. There would be a ton of different tables and only your office's specialty would be accessible by your office.
This is just chrome holding all your passwords. They're collocated in a higher layer, the browser, and then every site uses only the ones it needs (and can only access the ones it needs).
This would make the patient experience easier, would make referrals easier, would make everything easier.
You could also do some machine learning once all the data was collocated. I guarantee there are correlations we don't see yet because all the variables are never observed together.

Jinja can understand **kwargs, but the syntax is a little bit different than python: https://medium.com/python-pandemonium/using-kwargs-in-flask-jinja-465692b37a99.
Supercontest.

Week 4 did not become available at 10am PT, which would have been the case if I had messed up the timing (because that's 5pm UTC).
Restructured almost all of core/results, and its corresponding usage in views, to be sequential. Query for something, pass it to something else, sort it, calculate from it, etc. It's a much clearer, layered approach now. The analytic functions are modular, no queries are being repeated. Establishment of the necessary information happens FIRST, in the route, in the highest level. It's much easier to see duplicates now.
This modularity made my query footprint a lot smaller. The all-picks view would query for all users but only one week, whereas the lb/graph would query for all users for all week. They used the same machinery underneath, but the latter called it in a loop. Now it queries all at once and reformats the data as desired after. I remember thinking about this inefficiency when I first wrote it, but didn't have time to optimize. I do now.
Commented out all the tests so that I can run test-python to completion as it should. All no tests run, it's still gives more information to add this properly to my dev workflow (it compiles requirements and such).
Tested and work. App is a lot snippier, and FDT shows a lot fewer queries (like 10 max, for the worst route that has to query everything).
Updated to harner's gif. Played around with the background css styling. There are two important ones: background-repeat and background-size. "space" will keep the original size but fill in gaps with whitespace, "round" will distort the image within a slight margin to fill the gaps (both for repeat). For size, you can basically stretch it or keep it the same, fit in parent, etc. Looks good now. Fits one across on mobile, and repeats on larger viewports.
Made the app gracefully handle the time period between 5pm PT (week_start) and when westgate posts the lines, if there's a delay. It conditionally shows messages, and even omits the table.
The heaviest (time-wise) route in the main navs is /graph. The heaviest league is the free one, because it's the least filtered. And the heaviest season is any full one. Therefore, the current view that takes the longest to load is https://southbaysupercontest.com/season2018/league0/graph.
Removed wednesday from RESULTS_DAYS and just added a single commit_scores call within commit_lines, so it initializes properly.
Added queries.is_user_in_league and g.user_in_league.
Fixed a few things to make the new "return empty but valid results for new lines and new picks" work.
Confirmed that newly committed lines from westgate enter the db with timezone info, and timezone = utc (the rest were manually added to the migration - I wanted to make sure it was consistent.
Did the usual wed deployment.

If you follow a ctag in vim, don't edit it there. It won't save by default. This is just for looking.

Tuesday, September 24, 2019

To remove from a collection that represents a many-many relationship, just delete that list element and commit. You could also wipe it all by doing something like

user = db.session.query(User).filter(x).one()
user.leagues = []
db.session.commit()

I docker-compose downed the 3 bmahlstedt.com containers on the droplet so the supercontest can have the full resources of that tiny machine. I'll move bmahlstedt.com elsewhere. I'll have to remember to bring it back up for interviews, etc.
Cooked the soaked garbanzo beans. Scalded my hip by splashing boiling water directly on it. Was bad. Blistered up and got all oozy.
Bought bulk charcoal on amazon. Wasn't prime shipping, but it was $45 for 4x18lbs. Briquettes for $1/lb is usually a good deal - this is half that.
Big fresh order. $75. Should be good until tough mudder tahoe trip.
I should tweet this:

"[B*tches] aint leavin til 6 in the morning" - Snoop Dogg, Gin and Juice
"Parties dont stop til 8 in the morning" - Jermaine Dupri, Welcome to Atlanta
what kind of latenights are these rappers getting into

SQLAlchemy obviously does a join for all relationships. For many-one, it creates a member and for many-many it creates a collection. You can filter against these directly through the ORM, which is nice.

Use has() for member (many-one), like .filter(League.season.has(season=X))
Use any() for collection (many-many), like .filter(League.users.any(email=X))
HAS and ANY are called column comparators. There are a bunch: https://docs.sqlalchemy.org/en/13/orm/internals.html#sqlalchemy.orm.properties.ColumnProperty.Comparator.

Went to costco to buy beef ribs.
Interesting, but not surprising. Finishes get more likely as you go up in weight class:
Supercontest.

The league change alone was very simple. I only had to pass the league ID to like 5 places, and then add it as a filter to like 3 queries. This is due to the efficient layering of the query structure.
Confirmed in the league deploy that the flask monitoring dashboard data persists across versions now.
I could theoretically remove the entire dbsession.joins module now and just query on the backref relationships, but it's slower. The eager joins are much faster than going through the orm for everything.
Made it so that if a user is in a non-free league for the requested season, the default is to display that. If you're only in free, it shows free.
Adding coverer to the Score table, collocated as closely as possible to the information that it depends upon - this makes more sense than putting it in the Line table. There's also no real reason that it needs its own table, it's not distinct enough. It will update in realtime, not waiting until the game is finished. This is because some views want to update colors before the games are finished, but other views want to hold info until status=F/FO (like the lb). The information is there, you can do either.

All of the same logic above for coverer in the score table is repeated for points in the pick table.

is_league_in_season should return True if every checked for the free league. Fixed.
Did the usual efficiency optimization. Looked at FDT, removed unnecessary queries, consolidated others. Saved a ton of time, benchmarks are back down.
Renamed season_week_blueprint to data_blueprint.
Today was pretty dumb. Reverted a lot of the changes. Overall, pretty underwhelmed by the inability to nest blueprints, conditionally handle values, set up url defaults, preprocess, etc. Sharing app state to build urls for navlinks should not be this hard.
Ended up using flask-caching to just memoize the calls that were repeated, rather than solving the problem directly in application logic.
I have it configured to SimpleCache, which is just a python dict with key/values. Not the most thread safe, but should be fine.
Checked the FDT profiler to see if not just any sqlalchemy queries were being repeated, but python functions as well. Sorted by cumulative time, then calls, to make sure that nothing from my dist (supercontest.*) was being called a lot and taking a lot of time.

Monday, September 23, 2019

The droplet was slogging along.

Ran docker system prune --all, reclaimed 7.5G.
There is only one core, it's usually railed at almost 100% when deploying, handling a request, etc. It idles at 1-2% otherwise.
There's only 1G RAM. There's about 24G disk space. 0 swap.
Tried moving the wheelbuild from the intermediate container to a simple install in the final container. Still would hang.
sudo reboot. Wanted to restart the docker service but just did a whole system update anyway.
(unrelated) cleared the release updater with:

sudo truncate -s 0 /var/lib/ubuntu-release-upgrader/release-upgrade-available

sudo apt update && sudo apt upgrade
sudo apt full-upgrade
sudo apt autoremove
`sudo update-rc.d docker defaults` to make docker start on boot.
After system restart, mem went down to about 200M, 20%. It was at about 80% before, even before building the image.
Tried rebuilding the image for supercontest, worked this time. Looks like it was just system overload (I'm straining this tiny machine). A restart/upgrade every now and then won't kill us.
docker-compose restart supercontest-app-prod clears a decent amount of mem.
docker-compose down before running the new build also clears a lot of space.
The flask monitoring dashboard has a db backend, obviously. It's only 4M right now.

/proc/sys/vm/swappiness is how aggressively your system will use swap. It's configurable. 60 is the usual.
Pruned my laptop's docker system as well. Reclaimed 21GB.
Lost both fantasy games, both pretty badly.
Supercontest.

Added a persistent (named) docker volume for the flask monitoring dashboard. It was getting wiped new on every deploy. Confirmed it stayed.
Added ipython to the dev image so that the manage.py shell opens it by default.
Did some backlog cleaning and milestone organization. Issue count is in a good place.
Centered the headers on the rules markdown. Swapped the picks/points column on the all-picks view so the colored points cells would stand out, not adjacent to the full main table.
With Monday, Sunday, and Thursday games combined, there are ~18 hours of active scoring during the week. This is a little over 1000 minutes. If I fetch the nfl scoresheet xml once a minute, that's 1000 requests a week. Totally handle-able. With our current user base of ~50, and each user opening the app (or different views within it) about 10 times a week, we're already at half that traffic. Much better to control this and keep it static around 1000, rather than proportional to our user base as we scale.
With a backref or backpopulating relationship in sqlalchemy, graphene will break unless you name your SQLAlchemyObjectType classes distinct from the model name. I.e., Week becomes WeekNode.
Relationship changes are just in the ORM. They don't change any of the raw db, no sql, no migration/upgrade required.
Substantially cleaned the makefile and admin.md.
Added backrefs for all existing FKs in the models. There's a lot more information now!
I don't think you can filter on a backref with graphene. Still, it's useful to be able to see all the info (basically a join) in graphiql.
The automigration for the table creation of league and league_user_association was easy. The only manual changes to the migration file were the creation of the 2018 and 2019 paid leagues, then adding users to them.
To find all the users from 2018:

from supercontest.dbsession.joins import join_to_picks
all_picks = join_to_picks()
results = all_picks.filter(Season.season == 2018).distinct(User.email).all()
emails = [user.email for _, user, _, _, _, _ in results]

In the migration file, I create the league table and the league_user association, then I find all the users and add them appropriately. This migration was pretty cool.
Added full league functionality.
league0 is the free league. It's not an actual league row, it's just "don't do any filtering for league". It's the same as the app is today, before the league change. This allows users to play for free, any year they want.
The url_for(request.url_rule.endpoint, *args)) calls are layered. The season navs and league navs are lumped. They pass season and league, because they're the top level. The week navs pass season, league, and week, because they're the lowest. The information beneath (like what tab you're on: matchups or lb) is passed through request.url_rule.endpoint. You do not need to provide any lower information; ie league_navs don't have to pass week or main tab, and main navs don't have to pass week.
The nav_active js obviously matches the element's id to the route, not the displayed text. This allows you to show whatever you want while doing unique endpoint-nav matching.
Use g.<attr> in the templates, that's what it's for. The redefinition to variables in layout.html is for javascript only, not use in html.
Rewrote the url_defaults and value_preprocessor to be MUCH clearer about the actions each is taking, why, and when.
The hardest part of the league change was handling all the navs and links moving everywhere properly. This is a little easier in a node app with something like react/redux, where you have a full state container.
Did some cool conditional behavior on the matchups view. If a league was passed (via route explicitly, which flask must handle), it strips it and redirects to the leagueless matchups endpoint. The python side can parse and handle the values as args/kwargs at will, but if an url is fetched from a server, there MUST be a @route to handle it. You can't just url_defaults or url_value_preprocessor away it, you must add a @route to handle it first. Then you can redirect / add data / remove data as desired.

Turned on the activity overview (commits vs code reviews etc) for my github profile.
Ansible (even with -vvv and stdout_lines assigned to debug vars and such) does not print the stdout and stderr of the remote commands back on the host machine. If you wanna watch, ssh in and do it.
Reorganized and cleaned the kitchen, moving the machines under the shelves and removing the mat. Looks much better now. Cleaned the fridge/freezer too. Made new batch of oat milk. Starting soaking a large batch of garbanzo beans. Bottled the green tea hibiscus kombucha without a second fermentation. Threw away the scoby - I don't plan on making another batch soon.
Made another batch of protein bars. Just pecans, oats, and protein powder in this one. Delicious. Better than the tahini base.

I didn't think about it until after, but add some oat milk to the next batch. It will make them stick together better, allow you to add more protein powder, and impart a little taste.

Gbro didn't finish the 1/2 cup over the course of the day. I'll leave it at the same serving for now (1/2c once a day instead of twice). Online resources say that appetites vary, but a healthy cat weight is one where you can "easily feel the ribs". We're not close.
GitLab.

https://www.youtube.com/watch?v=nMAgP4WIcno.
Created an account, started to look through the full capabilities.
Combines Stash (SCM), JIRA (tickets), and Bamboo (CI/CD).
It's not an extension to github, it's a full alternative. It's a competitor.
You can edit in browser, merge requests (they call them the correct thing, not pull requests!), create boards, sprint plan, etc.
You get a lot of CI configuration right out of the box. It infers your language, creates a container, runs static analysis, checks licenses, runs security tests - all without you doing anything. You can, of course, customize all of this. They have advanced stages for dynamic testing, CSRF, and more.
It autocomments a lot of helpful information on the MR. Performance change, code quality deltas, etc.
CD has different environments, you can do partial deployments, more.
Microsoft bought GitHub last year for 7.5b.
I'm going to finish these few big tickets and then I'll migrate all my projects/repos over from github to gitlab.

https://www.programcreek.com/python/ is an awesome website. It basically just scrapes open source python projects for whatever you're looking for. In most cases, you'll get a working implementation of the string you're curious about.

Sunday, September 22, 2019

Remember to pass python lists through jinja with the |tojson filter for use in javascript. Same with bools. For dicts, dump them python side then pass with |safe.
Supercontest.

Added row count (rank, I guess, since it's sorted?) to the all-picks view. Makes it easy to compare to 45 (of the lb, which shows all users with season picks instead of week picks).
Added header with total pick counts for each team, and colored status of every game. This is the only place that has this info now; previously it would just highlight your picks on the matchups tabs, and then everyone's picks on this tab (which might have excluded teams; an often occurence).
App behaved well all day with the sunday rush, even with all the new changes (which were substantial). The live updated of points/possiblepoints for completed games was awesome. It ranked everybody dynamically, and was interesting to watch throughout the day.
Rewrote the `joins` module to properly use joins instead of subqueries. This is a lot cleaner. It removes all the `.c.` nonsense, and I don't have to be explicit (manually) about all the columns to forward through the joins. The only slightly annoying thing is that it returns the query results as a tuple divided by source table, so you have to unpack it to get the data you want. With the subquery structure, it was all flat so I could access the data directly with the column name.
Made the leaderboard sort with the same 3 keys as the all-picks view: total points, percentage, user id.
Added percentage at the top of the lb for each week, showing the performance of the entire league (theoretically should be close to 50).
Finished the graphql update. Fully got filters working. Didn't do joins. Uploaded a new client pkg to pypi with examples. Added twine instructions to ADMIN.md.

augmented-ui is a css kit you can include via cdn or npm pkg that transforms your site into cyberpunk: https://medium.com/better-programming/augmented-ui-cyberpunk-inspired-web-ui-made-easy-d463c0371144.
Foam roll on the legs = crucial for training.
Awesome website that does the same thing Ken's script used to do: https://www.fantasy-football-power-rankings.com/espn/1187979/2019. Power rankings, where it lists your record if you were to play every other team every week. Just like total points for, it is a more absolute metric of success without the random luck of head-head matchup order. Yahoo already reports this in your weekly recap, but ESPN doesn't, so this site provides it. https://github.com/JakePartusch/fantasy-football-power-rankings.
If joining the same table twice for multiple conditions, use sqlalchemy.orm.aliased. Same as a simple table table2 rename in sql.
Gonna feed gbro 1/2c twice a day instead of leaving big bowls out to self feed. This is slightly more than the recommended amount, but he's definitely overweight. I want to see how much he's consuming, then briley can adjust as he wants.
If you create a list in python with [x] * n, then they all point to the same object. If you update one of them, every element will mirror the same update.
Made another batch of tahini, but instead of using 100% sesame seeds, I used a ratio of about 80% sesame to 20% pecans, yielding pecan tahini. Incredible.
Made another batch of protein bars: oats, dates, protein powder, pecan tahini, banana, cinnamon, cacao powder. Incredible.
Read through most of the graphql documentation again.

Remember, it converts resolver names and such from snake_case to camelCase. I disably this during schema instantiation with auto_camelcase=False.
Within an ObjectType class, each resolve_x method corresponding to the field x that you define in the constructor.
They intentionally do not allow select *. Graphql is meant to be deliberate: fetch only what you need, not all cols. https://github.com/graphql/graphql-spec/issues/127.
SQLAlchemyConnectionField is the magic liason. Should map most of your entire model, all cols, field types, sortability, etc.
There is an extension to this that allows filtering: https://pypi.org/project/graphene-sqlalchemy-filter/. graphene-sqlalchemy can't do it right out of the box (simply). You'd have to add custom resolvers that pass args through query.filter() etc.
It's worth mentioning: the documentation surrounding this effort is horrendous. I have a very simple use case: take a few sqlalchemy models with foreign keys, allow flask graphql queries with filters on cols, for both direct queries as well as queries on joined tables. This common implementation should be trivial to implement, and it's not even close to that.
Filters: eq, ne, like, ilike, regexp, is_null, in, not_in, lt, lte, gt, gte, range.

Example: {seasons(filters:{season_start_lt:"2019-06-01"}){edges{node{id season}}}}. This would return just the rows from 2018 (lt = less than), and only the id and year number.

Datetimes are of the form "2019-11-01T00:00:00"

sqlalchemy has an `add_columns` which you can call after a join to create a more sql-standard select return table. https://docs.sqlalchemy.org/en/13/orm/query.html#sqlalchemy.orm.query.Query.add_columns. This can be used to get around the necessity of unpacking the multiple table results if you query a join. You can even add labels.

Saturday, September 21, 2019

TSS stands for Transportation Software Solutions. Let Stacey, Roubik, and Art know that my intentions are to move back to the Bay Area.
Remy and Katilin got sick on Monday too! It wasn't the prime rib, because everyone had that and only 3 of us reacted. I've also had it a few times since with no problem. Possibilities: dip, cheese, gravy? The latenight snack after stuff had been sitting out could have been it.
There's a paradox about predictability; that once something becomes predictable, entropy naturally makes it move away toward unpredictable again. I half-buy it. I'm a believer in determinism superseding free will in the longggg future.
Your body can actually burn an incredible number of calories while sedentary (comparable to rigorous physical exercise), if you are highly stressed/concentrated: https://www.espn.com/espn/story/_/id/27593253/why-grandmasters-magnus-carlsen-fabiano-caruana-lose-weight-playing-chess.
Agreed. Personal websites are wonderful tools. https://www.vanschneider.com/a-love-letter-to-personal-websites.
GitLab is a company I had seriously considered about 2 years ago: https://www.forbes.com/sites/alexkonrad/2019/09/17/gitlab-doubles-valuation-to-nearly-3-billion. They're doing very well, and my professional experience matches their mission absurdly well. Their IPO is in 1 year.
Chrome's devtools has a "device toolbar" that you can toggle on and off by clicking the little button that looks like a phone. You can specify any custom viewport, standard phones, and it will show your how your responsive view is displayed. Great for testing.
Hahaha AB released from the Pats. What a roller coaster.
Supercontest.

Added mx-auto to all nav cols so that they'd center on smaller viewports, tested in devtools. Looks much better.
The lb totals and percentages will now show as ints if they're whole numbers (even if source data is X.0). If they're floats, they'll continue to show as floats.
Webassets.

Moved all third-party cdn js to the toplevel template.
Moved all variables (that were passed through jinja from flask to javascript) to the toplevel template.
Bundled and minified all supercontest js, added in the toplevel template.
Cleaned some of the iteration in jinja, setting variables properly.
At this point, all templates are just templates. It's a lot cleaner. All variable injection is collocated, which is easier to understand as well.
Made all js in the bundle distinct. Some required more specific ID selectors. Some needed to check on array length. Some needed to condition on typeof <x> !== undefined. This whole exercise was a good practice in forcing explicit javascript.
Minifying with jsmin.

Package the service.

I don't just want to copy an ambiguous directory over to be able to use the app. It should have explicit boundaries. This will keep it slim, also.
Therefore, I made the service a self-sufficient package. It can't run without an accompanying database already existing, but the app layer itself is self-sufficient.
This was a pretty lengthy, iterative process to get to work.
MANIFEST only tells it what to build the dist with; not necessarily what to install. You have to be explicit about those as data files in setup.py.

Overall, working with bundled assets and wheel pkgs should make the app non-noticeably faster.
Closed https://github.com/brianmahlstedt/supercontest/issues/81.

Docker.

If you built an image successfully but it failed to run the start command or entrypoint for some reason or another, perform the following to test.

docker ps -a
docker commit <ID> test_image
docker run -ti --entrypoint=sh test_image

To force a rebuild, even if nothing has changed:

docker-compose build --no-cache <service>
Then you can start the service at will.

tar -xzvf (I can never remember anyway - this is a worthless comment).
Remember to build wheels in the same environment that you'll install them in. Not the same instance, but the same system. There are many underlying system libs that are baked into bdist wheels, and the build/deploy envs should be identical to prevent clashes for this.
find_packages() returns a list like this: ['supercontest', 'supercontest.core', 'supercontest.views', 'supercontest.dbsession', 'supercontest.commands', 'supercontest.models', 'supercontest.graphql']

Thursday, September 19, 2019

Supercontest.

Datetime, picks, leaderboard:

Updated all the datetimes in the database to pacific. Updated the database timezone to pacific.
Updated the url defaults. Season is current or max. Week is current or 17.
Added g.current_season next to the existing g.season (which is the requested season) and g.is_current_season. Same for weeks.
Switched everything to be perfectly clear: all datetimes in the database are utc, all westgate lines are written after conversion from pacific time, and all sqlalchemy reads convert to local.
Debug toolbar is helping a lot. Noticed that on RESULTS_DAYS, it was bloated by about 500ms due to excessive queries.score_exists/get_score calls. Consolidated all of them. You could even go further by conditioning commit_scores not just on RESULTS_DAYS, but on time as well to tighten the range in which the extra calls are made. Call fetch_scores protected by results_days, then only call commit if any of the statuses aren't P/F/FO.

Reduced total db queries for the endpoint by >30 and profile reduced by ~400ms.

Completed redefined the pick view and how it's sorted, including the core/results backend to calculate as such. Now returns number of picks, current total points, current possible points, etc. Looks very good now.
LB and graph only show completed games in the total now. I added a col for in-progress games so you call still check "what if it all ended right now" status.
Percentages are now based on the picks that you have submitted, not the total (5 per week).
Combined a lot of the route logic, keeping the endpoints as thin as possible. Most of them outsource to functions in the core.results module now.
Added tooltips all over leaderboard for submitted picks, help, more.
Universal color for a lot of things.
Renamed the matchups tab to your picks. Renamed the picks tab to all picks.
Graph and leaderboard are now bulletproof for completed games.
Added req compilation back as a requirement of the test step, and ensured that compreqs were installed into the testenv.

My make test-python recipe is heavy-handed; it will compile all reqs and recreate the testenv. On subsequent runs, just use tox.
Got the bandit tests running again. Pytest is still obsolete.
App behaved well over TNF.
Closed https://github.com/brianmahlstedt/supercontest/issues/84.
Closed https://github.com/brianmahlstedt/supercontest/issues/94.
Dashboard.

Added a conditional (on my account) navtab for the flask monitor.

Bools have to be filtered |tojson to pass through jinja from flask to js. They'll be properly typeof() = boolean in js.
Forcing NBA owners to change their titles to governors due to historical implications is the same false equivalence as telling a German that they can't use ovens. It's offensive. It's irrelevant. Race is being used as glue to compare situations which have absolutely no relation. It's not just a "doesn't affect you" situation - it's discriminatory, and should not be ok to concede. Finding these parallels with your lens of virtuous sensitivity is not progressive; it's an overshot of true equality, which slows resolution (if it doesn't destabilize entirely).
Nested dicts are fine to pass through jinja's |tojson filter for use in js.
Fed mouse. First time she's ever been snappy when I opened the terrarium. Used a broomstick to move her to the feeding tank.
My digital ocean droplet is in Santa Clara.

Supercontest.

Merged Spencer's PR to add req-compile (qer, sx-setuptools, pip-tools) to generate a compiled-requirements.txt for the package. https://github.com/brianmahlstedt/supercontest/pull/93/files.
Created new ticket for all datetime behavior: https://github.com/brianmahlstedt/supercontest/issues/94.

convert_date is only used in one place now: BEFORE it is written to the db.
start and end dates were added to all seasons and weeks.
This is a LOT cleaner for all the timing-related pieces of the app.
Successfully migrated and updated. Need to update the app for the new models tomorrow.

Added email tooltip to the picks view, just like the lb. Added notes to the top to explain.
Bugfixes for the new lines/picks to work with the huge model changes of last week: https://github.com/brianmahlstedt/supercontest/commit/8e27aafeca13c11c2e741a51d435f7482e4364ff.

Not too bad. I intentionally saved these for wed night local testing.

Ran the deploy, new banner, week 3 lines, backups.

Followup appointment with PCP. Got the referrals approved for both ortho and for PT.

The x-ray results:

Soft tissues: Unremarkable.
Bones: No acute fracture or subluxation. No lytic or blastic lesion.
Joints: The joint spaces are preserved. Articular surfaces are unremarkable.

So x-ray is fine. I'll probably need to do an MRI, since this would imply that the muscle is the source of injury.
Called the ortho place, made an appointment in Van Nuys for next wednesday.
I'm going to hold off on scheduling the PT until after the ortho evaluation is done. Stim, stretching, strengthening, etc - I can do that all on my own. The referral won't expired for 3 months, so I can schedule PT after ortho if need be.

Got adjusted at chiro (bought the 3/$90), then went to the new Chick-Fil-A.
For the last time: snake_case and camelCase and PascalCase.
Reminder for ISO8601: https://www.cl.cam.ac.uk/~mgk25/iso-time.html. All dates should be in the format YYYY-MM-DD.
A pipe | in a makefile is for order-only prerequisites. If you change any of those files, the parent recipe does not deem them necessary to rerun.
Postgres.

To get datatypes:

select column_name, data_type from information_schema.columns where table_name = '<mytable>';
select pg_typeof(<mycol>) from <mytable> limit 1;

I was very pleasantly surprised with postgresql's ability to internally parse timestamps. It correctly converted the type 'text' to 'timestamp with time zone' for every single datetime string from westgate. I thought I would have to use the python logic I had written to do this, but it was dead easy using sql in the migration instead.
lag() is a window function that allows you to update a row based on a previous row.

You can use direct sqlalchemy inside of an alembic migration if you want. This might be easier to (add, modify, etc) your data in python rather than doing it in sql. https://stackoverflow.com/questions/13676744/using-the-sqlalchemy-orm-inside-an-alembic-migration-how-do-i/
KFC is rolling out a chicken sandwich that uses glazed donuts instead of buns: https://www.cnn.com/2019/09/17/us/kfc-donut-chicken-sandwich-trnd/index.html.
Remember https://www.pngtosvg.com/ is the right place for free online vector conversion with color.
Just print(locals()) to show all args/kwargs passed to a function.
Changed my default text editor for git to vim: git config --global core.editor "vim"