Disclaimer. As anything else written in this blog, this article is for informational/recreational purposes only. It is not to be relied upon as a professional opinion whatsoever. For financial advice please seek certified professionals.

So. Every investing book out there for the past few decades is singing the same song. “Invest in an ETF that’s following a diverse index”.

The reasoning behind it is sound. Most indices are by nature diverse. E.g. SP500 is tracking the top 500 large-cap U.S. equities. It’s like a pointer that always points to the best of the best. By definition.

Also by sheer number alone it guarantees diversification which is the holy grail of risk-mitigation when it comes to investing.

And there’s more. Numbers are crystal clear in showing that indices are growing ever since their inception. They essentially are following the evolution of the economy they index and unless we’re talking of very specialised pockets of the economy the economy as a whole has been growing. It has had its hiccups (e.g. COVID) but every time it bounces back stronger and always trending upwards.

All of this is legit.

So, the prudent move for any investor with some brains in his head is to invest in indices. And the way to do that is through ETFs that is mirroring indices. Choose a broad-enough index (e.g. SP500), then choose an ETF based on that index and start buying shares of that ETF. Actually automate this even more by having some of your monthly salary purchase some ETF shares on a monthly basis. Automatically.

This has been extremelly popular for at least a decade now. And has delivered well. For example one of the biggest (i.e. popular) world ETFs,iShares Core MSCI World UCITS ETF USD (ISIN: IE00B4L5Y983), has delivered +521.52% since 2010. That is beyond amazing.

That is beyond amazing.

Enter Apple (NASDAQ: AAPL). At the time of this writing it constitutes 4.79% of above ETF holdings. After that is NVidia (NASDAQ: NVDA) with 4.75%. And so on with Microsoft, Amazon, Meta, Alphabet, Tesla. 69.19% of this ETF is invested in US companies. The next national market being Japan with 5.46%. So. Evidence number 1. Above ETF despite being a “world” one is almost by 70% a US invested ETF.

Now there seems to be a slight problem with US stock market. Take Apple for example. At the time of this writing its stock is trading at 254.49$ with a Price to Earnings (P/E) ratio of 41.86%. Now if we are to give any credit to fundamentalists, that arcane sect with members like Warren Buffet, a good P/E ratio for a stock is below 10. Some would go as far as 10-20. But 40ish is not something they’d stomach. That is to say that Apple stock seems way overvalued. The numbers look like that for all the big names that constitute the “world” ETF. So. Evidence number 2. Most of the holdings of the “world” ETF is overvalued stock according to fundamental analysis.

Now, and you’re not going to like this, there seems to be a systemic reason for this overshoot. And it’s not speculators that buy more Apple stock, despite their knowning of it being blown up disproportionaly, just to sell it a bit later for a profit. Let’s look who’s owning Apple. Here’s it https://finance.yahoo.com/quote/AAPL/holders/. Top holders is Vanguard and Blackrock with 8.91% and 7.23%… These happen to be the top issuers of ETFs. The example ETF we reffered at (iShares Core MSCI World) is issued by Blackrock. So. Evidence number 3. ETFs are already the biggest holders of the biggest companies indexed by the indices that these ETFs are mirroring.

Now let’s see what happens when you automatically save 100$ every month and direct them into the above ETF. 4.79$ of these are buying some Apple. Why? Because by definition that’s how ETF nvests your money. Mirroring the MSCI index it will invest to the companies in the index according to their weight on the index. Essentially your automatic money is voting for the current state of things. This makes the current state of things stronger. Apple stock is going up. Automatically. Your money won’t care if for example Apple went crazy and announced that it will ditch computers and start selling fruits. It will still buy Apple. Untill something else pushes Apple stock out of the index. Do you see it yet? ETFs are not mirroring the indices anymore. By their sheer weight and automatic nature they actively affect the indices. They are a major player in themselves and they are overheating the stock market. They are not mirroring a bubble. They are driving the bubble.

Now. Nobody knows the future but usually the markets correct themselves. At least they have done so for as long as they exist. I’m afraid that index-based ETFs will be one of the major if not the major culprits of the next market crash. And that’s hugely ironic because they indeed are the safest and most conservative investment vehicle. As long as we don’t jump on them en mass.

Very informative and comprehensive in its scope. Highly recommended to anyone building or tending modern distributed systems.

Amongst the highlights that stuck with me:

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together.—Doug McIlroy

It is my humble opinion that a huge portion of what is to be said of good system design has already been said in the way Unix was designed and a lot of what is bubbling up now and then as silver bullets are in fact bits and pieces of that early philosophy taken out of its grander context. That very quote about doing one thing well for instance was then brought into Object Oriented world as Single Responsibility Principle and, along with other similar maxims, were burrying more basic tenets under overengineered/unmaintainable designs that they were supposed to alleviate. See next quote for an indication of what might have gone wrong there. Microservices as a trend were initially going against complexity -I am a proponent of the line of thinking stating that a Microservice should be minimal- but in practice I see a lot of them degenerating to a “micro”monoliths.

Smalltalk is not only NOT its syntax or the class library, it is not even about classes. I’m sorry that I long ago coined the term“objects” for this topic because it gets many people to focus on the lesser idea. The big idea is ‘messaging’.—Alan Kay

That, together with immutability, is probably the best tools against complexity. I’m very happy to see newer tooling such as Golang, investing in this idea. (Erlang has been doing so for decades now with extraordinary results in terms of stable and scalable designs).

one Microservice is no Microservice — they come in systems. Like humans they act autonomously and therefore need to communicate and col‐laborate with others to solve problems—and as with humans, it is in collaboration that both the most interesting opportunities and chal‐lenging problems arise.

Ignore this and you’re set for chatty, inefficient microservices that won’t be able to scale.

You have to start by looking at the data and work with a domainexpert to understand its relationships, guarantees and integrity con‐straints from a business perspective, exploiting reality.This often includes denormalizing the data. Continue by definingthe consistency (transactional) boundaries in the system, within which you can rely on strong consistency. Then you should let theseboundaries drive the design and scoping of the Microservices. If youdesign your services with data dependencies and relationships inmind it is possible to reduce, and sometimes completely eliminate,the coupling of data

Data is more often than not the elephant in the room. Get the data model and sharing wrong and you might bring down the most scalable architectures.

TDD is basically programming by unit tests. Say you want to write a method XXX with inputs A, B and result C. Instead of going ahead and implement the method you first write a unit test that is testing the method:

def Test_X(self):
    # this fails without the correct XXX implementation
    self.assertEqual(wanted_output, XXX(a, b))

Now, with XXX actually missing, that test will fail untill you implement XXX and you implement it correct at least as far as the unit test is concerned.

It is an appealing idea as it gets the unit tests at the central place in the whole process. This combined with the wide adoption of agile methodologies during the past decade or so explains the hype about it.

But if there is one silver bullet in engineering is that there is no silver bullet in engineering.

TDD’s main, albeit somewhat hidden, assumption is that you know beforehand which method(s) you need and what their signatures will be. That, though, is quite rare in practice as Software is in a constant state of change especially in an agile environment where TDD is supposed to fit in best.

This is even more true in the begining stages of Software creation where one does not know exactly how he’ll go about to solve a problem. Various hindrances along the way might, and often do, require big changes in code path, data models and methods. Doing TDD along these stages means a lot of wasted effort as you refactor your code base again and again. And wasted effort is frustrating and unproductive. I think it’s way more productive and better developer experience if at least during the early stages you went straight to implementation and you only invested in unit tests after the code base has started to be a bit less of a moving target, which usually happens after a few iterations. Once you get there you could switch to TDD if that’s your cup of tea and maybe you should as it makes it more difficult to forget unit testing new code.

Above hybrid methodology is what I’ve seen in practice and I think it strikes a very good balance between productivity and tests coverage. I don’t know if it goes by any name so lets just call it Tests After Code (TAC).

Note: There’s another catch with TDD, namely a false sense of coverage. You see if you TDD you usually write one test to cover the yet non-existent method. There’s nothing stopping you for writing more but given the hassle we just described we can assume that under pressure most code authors would stick to one test. You then write the method and it passes the test and that’s that. Only it isn’t. You see your method might be correct in one way but might err in many other ways. Which is why usually people write more than one tests for each method. Yet since the weight in TDD is in passing that initial gatekeeper test I wouldn’t be surprised if that’d also be the only test covering your method.

In these cases what you might end up doing is run the whole program once and keep the output you want in a file. You then copy that file into your test resources and you load it up in your unit test in order to compare it with the current output (and fail the test if the two differ).

That is fine and it works for some time. Untill you change something and the output is expectedly different. In which case you either need to dive into the resources file and fix it up or repeat the above produce-save-copy process.

Instead you can employ a simple trick to have the tests actually do that for you.

Assume next typical folder structure of a (python) project:

proj/
    stuff/
        __init__.py
        stuff.py
        other_stuff.py
    tests/
        __init__.py
        test_stuff.py
        test_other_stuff.py

And say that your long output function lies in stuff.py and looks like:

import os
from jinja2 import Template

def render(templ, conf):
    """
    Renders the jinja template in file templ according
    to the settings in dictionary conf
    and returns rendered result (str)
    """
    template = Template(open(templ).read())
    return template.render(conf)

Then inside the relevant test in test_stuff.py you can have something like:

from .__init__ import record_tests

def test_render(self):
    test_resource = 'resources/out'
    template = 'template.jinja'
    config = {'name': 'val', 'other_name': 'val'}
    have = render(template, config)

    # if in record_tests mode capture what we got in resources file
    if record_tests and record_tests == "yes":
        with open(test_resource, 'w+') as outf:
            outf.write(have)

    with open(test_resource, 'r') as wantfile:
        want = wantfile.read()
    self.assertEqual(have, want)

So when record_tests is set to “yes” the output of the tested function is saved as the test_resource. This saves us the trouble of going over the produce-save-copy process.

BUT

When record_tests is set to “yes” the test will always pass thus rendering it useless. So it should be set in a manner that won’t accidentally leave it “on” inside your CI pipelines.

An easy way to do that is to have it getting a value off your environment and do that once and for all tests. That’s what the funny import

from .__init__ import record_tests

does. And the value is set from the environment in the tests\__init__.py file:

import os

record_tests = os.getenv("RECORD_TESTS")

Assuming that your CI engineer (which is probably you wearing your CI hat) has a good overview of the env vars that live in his pipeline, we can assume that the conspicuous variable RECORD_TESTS won’t be set there or at least it won’t be set by accident.

Back to the initial senario of you making an intentional change to your function. You can now run your tests once locally with RECORD_TESTS env var set -thus recording the new outputs- and then rerun your tests to make sure everything looks good.

$> RECORD_TESTS=yes pytest
...
$> pytest

The first command sets the env var just until the completion of the command so you can rest assured that the env var won’t exist after it. Thus, any subsequent tests run is done in the normal non-recording mode. Having the changes in and the updated tests passing you can push all the changes to your code repo and go for a nice walk.

• Count specific elements in list: len([x for x in mylist if x==value])

• Create a 2D matrix: Matrix = [ [0 for x in range(cols)] for y in range(rows)] Ref. element in row i, col j: Matrix[i][j]

• Initialize a list with a map function: map( lambda x: x+1, range(3) ) ==> [1, 2, 3]

• a list of n zeros: [0] * n

• exchange values of two vars: a, b = b, a

• *, **

git

• Squash (already pushed) 3 commits: git rebase -i upstream/onboarding~3 onboarding (watch the tilde ~) (replace pick with fixup) git push -u upstream +onboarding (watch the +) more: https://stackoverflow.com/questions/5667884/how-to-squash-commits-in-git-after-they-have-been-pushed