Stream: t-compiler/wg-polonius

Topic: performance


nikomatsakis (Jul 03 2019 at 10:24, on Zulip):

Hey @Albin Stjerna -- in polonius#109, you mentioned perf was abysmal?

Albin Stjerna (Jul 03 2019 at 11:01, on Zulip):

@nikomatsakis It might have been a bit of an overstatement, given that I haven't even measured it, but I mainly meant the delta of performance introduced by the changes surely must be, given that I am for every variable walking the entire MIR and generating facts, etc

Albin Stjerna (Jul 03 2019 at 11:01, on Zulip):

It's not a problem yet, especially as we're not going to keep it

nikomatsakis (Jul 03 2019 at 11:01, on Zulip):

ok ok

Albin Stjerna (Jul 03 2019 at 11:01, on Zulip):

But it seems that the liveness calculations in general does play a big role in the solve-time in my benchmarks

nikomatsakis (Jul 03 2019 at 11:02, on Zulip):

yeah, we need to investigate -- among other things, we don't actually need to compute liveness for all variables

nikomatsakis (Jul 03 2019 at 11:02, on Zulip):

this was something we spent some time tuning in rustc

Albin Stjerna (Jul 03 2019 at 11:02, on Zulip):

I can imagine

Albin Stjerna (Jul 03 2019 at 11:03, on Zulip):

This is the heatmap from 789 518 functions of 1738 crates:
pasted image

Albin Stjerna (Jul 03 2019 at 11:03, on Zulip):

(where it is among other things obvious that my analysis on the CFG gave no useful information)

Albin Stjerna (Jul 03 2019 at 11:03, on Zulip):

It's correlation vs normalised runtime

Albin Stjerna (Jul 03 2019 at 11:04, on Zulip):

In particular, the initialisation and liveness facts are really hot

Albin Stjerna (Jul 03 2019 at 11:04, on Zulip):

But yes, More Analysis Needed™

Albin Stjerna (Jul 03 2019 at 11:04, on Zulip):

(I had some errands, but I'll finish the PRs for review soon!)

Albin Stjerna (Jul 03 2019 at 11:05, on Zulip):

Oh, and these are histograms for the solve-times: pasted image

Albin Stjerna (Jul 03 2019 at 11:06, on Zulip):

As you can see, very short solve-times completely dominate

nikomatsakis (Jul 03 2019 at 11:06, on Zulip):

one other thing -- in the PR, there is a CLI flag for disabling the region_live_at facts -- do we need that? can we just remove said facts altogether?

nikomatsakis (Jul 03 2019 at 11:06, on Zulip):

As you can see, very short solve-times completely dominate

what is the range of samples here?

Albin Stjerna (Jul 03 2019 at 11:06, on Zulip):

For solve-times?

Albin Stjerna (Jul 03 2019 at 11:07, on Zulip):

Hmh, I guess we could, but then we can't do the verification anymore

Albin Stjerna (Jul 03 2019 at 11:07, on Zulip):

But my plan was to remove it in the long run

nikomatsakis (Jul 03 2019 at 11:07, on Zulip):

yeah, I wondered about that, it also seems fine to keep it for verification purposes

nikomatsakis (Jul 03 2019 at 11:07, on Zulip):

I'm just shooting for simplicity :)

Albin Stjerna (Jul 03 2019 at 11:08, on Zulip):

I'm fine with eventual simplicity :)

nikomatsakis (Jul 03 2019 at 11:08, on Zulip):

For solve-times?

right I mean what are the "Test cases"

Albin Stjerna (Jul 03 2019 at 11:08, on Zulip):

Ah, the test cases are all the facts generated by analysing 1738 of the most popular (downloads on crates.io and stars on github) crates

Albin Stjerna (Jul 03 2019 at 11:08, on Zulip):

I have more, but ran out of hard disk space to store the facts

lqd (Jul 03 2019 at 11:27, on Zulip):

... generated by analysing 1738 of the most popular (downloads on crates.io and stars on github) crates

do you happen to remember how many of those actually failed to build ?

Albin Stjerna (Jul 03 2019 at 11:32, on Zulip):

No, I didn't collect that data. Hmm, I even think a vast number of those succeeded building

Albin Stjerna (Jul 03 2019 at 11:32, on Zulip):

Because I may have only included those

Albin Stjerna (Jul 03 2019 at 11:32, on Zulip):

Yes, I think so

Albin Stjerna (Jul 03 2019 at 11:33, on Zulip):

That's not great now that I think about it

lqd (Jul 03 2019 at 11:34, on Zulip):

oh it makes perfect sense to do so for benchmarks; I was just trying to gather new data by proxy, of possible Polonius bugs :)

Albin Stjerna (Jul 03 2019 at 11:34, on Zulip):

Oh, cool; the heatmap changes when you filter out the (arguably very few) functions above 10 000 edges in the CFG

Albin Stjerna (Jul 03 2019 at 11:35, on Zulip):

(almost all of them end up on 10 000 edges or fewer)

Albin Stjerna (Jul 03 2019 at 11:36, on Zulip):

Then the CFG doesn't matter at all, and the number of variables and var_defineddominate along with the other var_... tuples

lqd (Jul 03 2019 at 11:37, on Zulip):

I can't wait for our correctness/testing/validation to be improved so we can see how many of those edges are actually needed. I have coined the term "differential liveness" and am looking for a reason to use it when we possibly bring CFG compression back from limbo :)

lqd (Jul 03 2019 at 11:41, on Zulip):

(deleted)

lqd (Jul 22 2019 at 02:26, on Zulip):

@Albin Stjerna I might need some new benchmarks more representative than clap, could you tell me which of your 10000 test crates looked interesting to profile ? (eg. the slowest running or the ones with the biggest overhead; or maybe also the ones where the overhead is big without counting the fact generation time) thanks in advance :)

Albin Stjerna (Jul 22 2019 at 16:34, on Zulip):

Albin Stjerna I might need some new benchmarks more representative than clap, could you tell me which of your 10000 test crates looked interesting to profile ? (eg. the slowest running or the ones with the biggest overhead; or maybe also the ones where the overhead is big without counting the fact generation time) thanks in advance :)

I don't measure fact-generation time (but I probably should). In fact, I don't even record specifically which repositories have timeouts or memory-outs. But I can get you a list of long-running repositories or repositories (in terms of Polonius runtime) with large inputs!

Albin Stjerna (Jul 22 2019 at 16:36, on Zulip):

The worst recorded runtime (a severe outlier by more than a factor of 10) is from this repo: https://github.com/thuleqaid/rust-aaplus

Albin Stjerna (Jul 22 2019 at 16:37, on Zulip):

Ah, so that's a large const array, surprise

Albin Stjerna (Jul 22 2019 at 16:37, on Zulip):

Let's see if I can find something a bit more interesting

Albin Stjerna (Jul 22 2019 at 16:47, on Zulip):

Fascinating, about 64% of the functions in the data set have no loans at all

Albin Stjerna (Jul 22 2019 at 16:55, on Zulip):

@lqd Hm, this method has the worst recorded runtime: https://github.com/rodrimati1992/type_level/blob/master/derive_type_level_lib/src/typelevel/compiletime_traits.rs#L29

Albin Stjerna (Jul 22 2019 at 16:58, on Zulip):

(deleted)

Albin Stjerna (Jul 22 2019 at 16:59, on Zulip):

Woops, I apparently managed to dump the whole file, not the head of the file

Albin Stjerna (Jul 22 2019 at 17:00, on Zulip):

Here's a CSV with the 500 slowest functions and their programs (=repositories)

lqd (Jul 22 2019 at 18:02, on Zulip):

@Albin Stjerna awesome thanks a lot

lqd (Jul 22 2019 at 18:02, on Zulip):

that will be super helpful, I’ll try some of them later tonight

Albin Stjerna (Jul 22 2019 at 18:03, on Zulip):

Let me know if you want actual fact files!

Albin Stjerna (Jul 22 2019 at 18:03, on Zulip):

I have fact files coming out of my ears

Albin Stjerna (Jul 22 2019 at 18:03, on Zulip):

Literally 400 GB

Albin Stjerna (Jul 22 2019 at 18:04, on Zulip):

But it was very good that you asked, because it made me aware of the no-loans outliers which had some very extreme values, and my data got much less messy after removing them

lqd (Jul 22 2019 at 18:05, on Zulip):

the OOMs would be interesting to know, and like the other case I mentioned earlier we can even know if it’s in the facts or Polonius itself

lqd (Jul 22 2019 at 18:06, on Zulip):

and also, the ones where Polonius and NLLs don’t give the same answer (I assume all these benchmarks must somehow compile today, any Polonius error would be... annoying :)

lqd (Jul 22 2019 at 18:07, on Zulip):

but until my PR lands, at the very least this can happen in cases where Polonius lacks some "killed" facts, and there’s probably other bugs of the sort elsewhere

lqd (Jul 22 2019 at 18:09, on Zulip):

(but if you got facts for most of those, I assume they didn’t OOM during fact generation)

lqd (Jul 22 2019 at 18:13, on Zulip):

64% with no loans oh wow

lqd (Jul 22 2019 at 18:14, on Zulip):

if there are no invalidations (when we’ve also dealt with illegal subsets relations) we can bail early :)

lqd (Jul 22 2019 at 18:15, on Zulip):

(but some of the work I’ve been doing should help there anyway so I’ll check some of those out as well)

lqd (Jul 22 2019 at 18:19, on Zulip):

do you have the data from which you’ve extracted the csv available somewhere ? I’m very much looking forward to reading your masters btw !

Albin Stjerna (Jul 22 2019 at 19:41, on Zulip):

do you have the data from which you’ve extracted the csv available somewhere?

No, I only have it in raw form otherwise, that is just the fact files in TSVs. Those are the 400 GBs of files. I do have some error logs which I think could be analysed to figure out what went wrong, but that's about it. From the perspective of my analysis, the CSV is the "raw" data.

I’m very much looking forward to reading your masters btw !

Aww thank you :)

Albin Stjerna (Jul 22 2019 at 19:47, on Zulip):

The runtime for the naive algorithm is...weird though. This is an attempt at correlating it to various inputs

Albin Stjerna (Jul 22 2019 at 19:48, on Zulip):

Looks almost exponential or something

Albin Stjerna (Jul 22 2019 at 19:49, on Zulip):

Also, even the less intelligent hybrid version pays off slightly

lqd (Jul 22 2019 at 19:57, on Zulip):

yeah the naive algorithm does a lot of compute and then throws most of it away

Albin Stjerna (Jul 22 2019 at 19:58, on Zulip):

But apparently it's worth it!

Albin Stjerna (Jul 22 2019 at 19:58, on Zulip):

That's the shocking part

lqd (Jul 22 2019 at 19:59, on Zulip):

From the perspective of my analysis, the CSV is the "raw" data.

yeah this CSV is what I was referring to as "the data", ie on the whole 10k suite

Albin Stjerna (Jul 22 2019 at 19:59, on Zulip):

Ah, ok, well that I can just upload

Albin Stjerna (Jul 22 2019 at 19:59, on Zulip):

err, maybe

lqd (Jul 22 2019 at 20:00, on Zulip):

no one else will probably read this, so I can tell you, I tried heavily filtering the variants, the speed on clap is insane :p

Albin Stjerna (Jul 22 2019 at 20:00, on Zulip):

ok, it's 500MB so maybe not

Albin Stjerna (Jul 22 2019 at 20:00, on Zulip):

What does "heavily filtering" mean?

Albin Stjerna (Jul 22 2019 at 20:00, on Zulip):

What did you filter in/out?

lqd (Jul 22 2019 at 20:01, on Zulip):

only tracking the loans which can have errors, only do the TC on the regions downstream from the borrow regions of the loans which can be invalidated

lqd (Jul 22 2019 at 20:02, on Zulip):

(kinda annoying wrt illegal subset relations, but as the DatafrogOpt variant couldn't compute them via subset we'd have to deal with that eventually, and I think I can do this filtering for good)

Albin Stjerna (Jul 22 2019 at 20:02, on Zulip):

Ah, ok. When do you do the filtering? When generating facts or in Polonius itself?

lqd (Jul 22 2019 at 20:03, on Zulip):

rn in Polonius

Albin Stjerna (Jul 22 2019 at 20:03, on Zulip):

Ah, ok

lqd (Jul 22 2019 at 20:03, on Zulip):

to be able to use the same infrastructure to share between the loc insensitive variant and the variants

Albin Stjerna (Jul 22 2019 at 20:03, on Zulip):

Here's the full CSV

lqd (Jul 22 2019 at 20:04, on Zulip):

except that I can also only filter for each analysis, and it does basically the same thing on clap

lqd (Jul 22 2019 at 20:04, on Zulip):

that's why I'd like to try on more inputs

lqd (Jul 22 2019 at 20:04, on Zulip):

thanks a lot for the CSV :)

Albin Stjerna (Jul 22 2019 at 20:04, on Zulip):

No problem! :)

lqd (Jul 22 2019 at 20:05, on Zulip):

I can send you the WIP if you want

lqd (Jul 22 2019 at 20:06, on Zulip):

(once you know what to do it's pretty simple anyway)

Albin Stjerna (Jul 22 2019 at 20:09, on Zulip):

Sure, why not

lqd (Jul 22 2019 at 20:16, on Zulip):

pretty effective on clap but we'll see

lqd (Jul 22 2019 at 20:17, on Zulip):

the locinsensitive pre-pass would just more heavily filter the possible loans

lqd (Jul 22 2019 at 20:17, on Zulip):

instead of all the invalidated ones

Albin Stjerna (Jul 22 2019 at 20:18, on Zulip):

That sounds like a good idea

Albin Stjerna (Jul 22 2019 at 20:18, on Zulip):

I wonder how well the input facts would compress, given that they are tab-separated text files

Albin Stjerna (Jul 22 2019 at 20:18, on Zulip):

Probably incredibly well

lqd (Jul 22 2019 at 20:19, on Zulip):

(I also have a prototype of sharing the pre-pass results)

lqd (Jul 22 2019 at 20:19, on Zulip):

haha yeah maybe :)

lqd (Jul 22 2019 at 20:20, on Zulip):

if it does well on other inputs, I wish I tried this a year ago...

Albin Stjerna (Jul 22 2019 at 20:21, on Zulip):

I really wish we had a robust benchmarking infrastructure of some sort

Albin Stjerna (Jul 22 2019 at 20:21, on Zulip):

Every time I think about something to optimise I see the wall in the factory at CERN where they made the gigantic dipole magnets; YOU CAN ONLY MAKE AS WELL AS YOU CAN MEASURE

lqd (Jul 22 2019 at 20:21, on Zulip):

maybe we can extend lolbench

Albin Stjerna (Jul 22 2019 at 20:22, on Zulip):

It seems defunct?

Albin Stjerna (Jul 22 2019 at 20:23, on Zulip):

I have a strong feeling the route of "improvise a set of Python scripts" isn't the right one :)

Albin Stjerna (Jul 22 2019 at 20:23, on Zulip):

Or, it absolutely was for shipping my thesis on time, but not for infrastructure

lqd (Jul 22 2019 at 20:23, on Zulip):

yeah anp doesn't have much time to work on it nowadays, but he did a bunch of work on it, to get stable numbers, gather hardware counters, etc

lqd (Jul 22 2019 at 20:24, on Zulip):

automate regression detection etc

lqd (Jul 22 2019 at 20:24, on Zulip):

he did a talk about it as well

Albin Stjerna (Jul 22 2019 at 20:24, on Zulip):

Cool

lqd (Jul 22 2019 at 20:35, on Zulip):

I wonder how well the input facts would compress, given that they are tab-separated text files

I think 10x easily (esp for the bigger ones, like clap), maybe it could be interesting to have them as zip files in the repo and the polonius binary could stream the .facts files

lqd (Jul 22 2019 at 20:36, on Zulip):

although I think clap's size has reduced a lot since you regenerated it in the liveness PR

lqd (Jul 22 2019 at 20:38, on Zulip):

I don't know if it's from rustc changes or something else (what version of clap did you generate those from ?) but I think the Naive variant processed the new ones around twice as fast as the old facts

Albin Stjerna (Jul 23 2019 at 13:09, on Zulip):

what version of clap did you generate those from?

The one in current crater, I think.

Yes, there definitely is a lot of interactions with other rustc changes affecting the Polonius inputs.

lqd (Jul 23 2019 at 13:10, on Zulip):

in crater or the old in rustc-perf ?

lqd (Jul 23 2019 at 13:10, on Zulip):

(the latter is the one which caused problems IIRC)

lqd (Jul 23 2019 at 13:28, on Zulip):

I'm having a bit of trouble reproducing some of the slower benchmarks you mentioned, maybe generating the facts has become quite slow

lqd (Jul 23 2019 at 13:33, on Zulip):

(probably when the liveness or partial initialization data is huge ? -- update: yes in this benchmark's case it was a lot of var_initialized_at_exit taking 10x the time)

Albin Stjerna (Jul 23 2019 at 15:53, on Zulip):

Hmm, it was probably the old one in rustc-perf now that you mention it

Albin Stjerna (Jul 23 2019 at 15:54, on Zulip):

Yes, probably. It's incredibly inefficiently implemented which might skew the results

Albin Stjerna (Jul 23 2019 at 15:54, on Zulip):

Means I will probably have to re-do the analysis

Albin Stjerna (Jul 23 2019 at 15:54, on Zulip):

Given that I'm currently working on removing that input

lqd (Jul 23 2019 at 16:17, on Zulip):

just to make sure I checked typelevel-compiletime_traits-{{impl}}[1]-to_tokens I can reproduce your 30+s seconds on the Naive variant, and indeed my latest rev works nicely on it

lqd (Jul 23 2019 at 16:18, on Zulip):

are the Naive/Hybrid/DatafrogOpt times in your CSV taken with rustc btw ? or by running polonius with -a ?

lqd (Jul 23 2019 at 16:23, on Zulip):

to make sure again, are these benchmarks only for code which has no errors under -Z polonius ? if that's the case I wonder if the LocationInsensitive returns errors in some of these cases, so we can see the difference it would make to share the results from LocationInsensitive to DatafrogOpt in the Hybrid variant.

lqd (Jul 23 2019 at 16:27, on Zulip):

(although it's possible the Naive variant is already getting good enough at finding no errors pretty fast - again, modulo subregion obligations; it may be time for some synthetic benchmarks as well)

Albin Stjerna (Jul 23 2019 at 17:48, on Zulip):

are the Naive/Hybrid/DatafrogOpt times in your CSV taken with rustc btw ? or by running polonius with -a ?

Just running Polonius! Specifically, I have --skip-timing --ignore-region-live-at

Albin Stjerna (Jul 23 2019 at 17:49, on Zulip):

Regarding errors, that's...harder

Albin Stjerna (Jul 23 2019 at 17:49, on Zulip):

But anything that returns a non-zero exit status on fact generation would have been discarded

Albin Stjerna (Jul 23 2019 at 17:52, on Zulip):

Or anything that returned something non-zero when doing cargo +nightly check with RUSTFLAGS set to -Zborrowck=mir

Albin Stjerna (Jul 23 2019 at 17:53, on Zulip):

Oh, I should add: times are recorded by Python itself and I'm taking the minimum over three runs

Albin Stjerna (Jul 23 2019 at 17:54, on Zulip):

It should be the same as the standard unix time command would report, but who knows how it would relate to the Polonius output

Albin Stjerna (Jul 23 2019 at 17:55, on Zulip):

Actually, if I ever re-do it, I should probably only run one time and in stead parse the output and use the times reported by Polonius, but I just wanted something I knew worked and not have to deal with wonky output or mismatches etc

lqd (Jul 23 2019 at 21:16, on Zulip):

ah ok, running the polonius binary will also contain the time to read the files on disk, there will be many avoidable clones of the facts, etc

Albin Stjerna (Jul 24 2019 at 08:22, on Zulip):

Yes, that's true

Albin Stjerna (Jul 24 2019 at 08:23, on Zulip):

Maybe we should put some work into a pure profiling mode, e.g. one that, say, outputs CSV directly to stdout

Albin Stjerna (Jul 24 2019 at 08:23, on Zulip):

(by "we" I mean "I")

Albin Stjerna (Jul 24 2019 at 08:23, on Zulip):

But there already is something like that for rustc, right?

lqd (Jul 24 2019 at 08:33, on Zulip):

yeah and that's what lokalmatador was working on, using the profiling systems inside rustc to time polonius

lqd (Jul 24 2019 at 08:34, on Zulip):

since they were already gathering some data I assume the general setup was completed

lqd (Jul 24 2019 at 08:35, on Zulip):

ah but they might have deleted their GH account, with the WIP branch mentioned here

Albin Stjerna (Jul 24 2019 at 13:41, on Zulip):

Oh, that's too bad

Albin Stjerna (Oct 23 2019 at 14:32, on Zulip):

Ok I should absolutely talk to Yannis about Polonius, I just need to catch him when he’s not being accosted by anyone else which is all the time

lqd (Oct 23 2019 at 14:33, on Zulip):

I think Niko talked to him about that at SPLASH last year :)

Albin Stjerna (Oct 24 2019 at 08:09, on Zulip):

He held a talk about how Datalog was the future of programming specifically mentioning some problematic join patterns that I'm quite sure are behind my abysmal liveness performance

Albin Stjerna (Oct 24 2019 at 08:23, on Zulip):

HA Yannis promised to have a look at my Datalog rules and see if he saw some Smart Things™️ I could do. He also seemed genuinely sad @nikomatsakis wasn't coming. :(

lqd (Oct 24 2019 at 09:48, on Zulip):

(I think I didn't mention it but I also worked on helping automate more of our datalog to datafrog pipeline specifically to implement the provenance graphs rules rewriting Yannis describes in the declarative debugging paper)

lqd (Oct 24 2019 at 10:14, on Zulip):

(oh and if anyone is interested in the specific talk, it's from a paper from a few months back and is already available on youtube)

Albin Stjerna (Oct 24 2019 at 13:51, on Zulip):

Wow cool!

Albin Stjerna (Oct 24 2019 at 13:55, on Zulip):

Also I of course didn’t ask Yannis to debug my code, I said hi, explained what I had been working on and asked for a book recommendation.

Albin Stjerna (Oct 24 2019 at 13:56, on Zulip):

I realised I sounded like a jerk

Albin Stjerna (Oct 28 2019 at 20:19, on Zulip):

I have been thinking about something. If the root of the performance problem with liveness and initialisation is that we over-compute, why don't we compute these relations top-down instead? I.e first compute a set of potentially interesting origins by joining with potential errors, then join that subset to their corresponding variables, and then compute liveness for those variables?

Albin Stjerna (Oct 28 2019 at 20:21, on Zulip):

And the same with initialisation of course; first compute a set of interesting move paths by finding accesses and reference-containing structs, and then determining the initialisation for those?

Matthew Jasper (Oct 28 2019 at 20:23, on Zulip):

We probably should. I think that we need to be a bit more clever to see large enough savings, and having everything fully implemented makes this easier.

Albin Stjerna (Oct 28 2019 at 20:42, on Zulip):

Ok, good, that means I haven't completely misunderstood everything :)

lqd (Oct 29 2019 at 00:09, on Zulip):

computing the set of potentially interesting origins with potential errors and then going up is the version of the Naive variant I sent you a couple months back, and which I've recently said made 20x on liveness on clap :)

lqd (Oct 29 2019 at 00:12, on Zulip):

and 30x on initialization on clap, but prevented from computing all init/move errors, since it was computing over only move paths belonging to variables having origins in their types into which potential erroneous loans could flow

lqd (Oct 29 2019 at 00:19, on Zulip):

(it will still be bottom-up unless you don't mean top-down/bottom-up in the datalog/prolog sense, and overall there are also eg the MST/DT transforms which hope to make a query's bottom-up evaluation need the tuples a top-down evaluation would need; and if you do mean top-down in the datalog sense then we'd need a different "solver", like an SLG solver, maybe chalk's)

lqd (Oct 29 2019 at 00:27, on Zulip):

until we have all we need implemented, the performance work and the completeness work kinda deadlock each other, for example: depending on how we implement illegal subset relations, the filtering we describe here could be undersirable (even DatafrogOpt already clashes with some of these different possibilities). now for this particular example, placeholder loans (the 3rd way to implement these errors) seem to allow for these optimisations so yay

lqd (Oct 29 2019 at 00:30, on Zulip):

and, probably, "all we need" doesn't include the currently nebulous plan of dealing with higher ranked subregions via a chalk-to-polonius lowering step :) (I hope)

Albin Stjerna (Oct 29 2019 at 16:23, on Zulip):

... unless you don't mean top-down/bottom-up in the datalog/prolog sense ...

I did, but I meant in the dumbest way possible

Albin Stjerna (Oct 29 2019 at 16:23, on Zulip):

(thank you for explaining!)

Albin Stjerna (Oct 29 2019 at 16:24, on Zulip):

I meant as in, do the joins in datafrog one at a time and compute them until fixpoint one at a time

Albin Stjerna (Oct 29 2019 at 16:24, on Zulip):

For some interesting tuples

lqd (Oct 29 2019 at 16:49, on Zulip):

(at a glance, I think killed and invalidates may be too similar, it's not about the same loans IIRC)

Last update: Nov 15 2019 at 20:05UTC