Andy Granowitz

Empirical explanation for Craig Wright's selfish mining Bitcoin bet

Thu, 12 Apr 2018 00:00:00 +0000

Last year, Craig Wright and Peter Rizun bet 1 Bitcoin on this problem:

Evan Klitzke has a good explanation of the problem and a theoretical explanation of why Peter is correct. As Evan explains, because “mining approximates independent trials of a random process” the expected time for an honest miner to find a block is always 15 minutes, regardless of how much time they’ve been trying to find a block. Each time a miner increments a nonce and computes a block’s hash, they have a nearly equal chance of finding the nonce that “gives the block’s hash the required zero bits”, to quote Satoshi. No progress is made no matter how many hashes the miners have performed. The time between mined blocks is memoryless.

But we don’t need to rely on a theoretical argument alone. In the bet, the honest miners control ⅔ of the hash power, making the expected time for an honest miner to find a block 15 minutes, as Evan explains:

T_expected = 10 / alpha

Alpha of ⅔ yields 15. To empirically show why Peter is correct using real Bitcoin block time data, we’ll need to adapt the problem. By looking at the entire Bitcoin network (100% of hashpower) instead of only a subset of miners, we should expect 10 minutes until the next block is found, no matter how much time it has been since the last block was found.

Let’s find out if the data supports this. BigQuery recently published all historical Bitcoin blockchain data. It’s an easy way to analyze Bitcoin data. I brought the data into R for analysis and visualization (full code below).

One simplifying assumption: in order to analyze time between mined blocks, I sorted blocks by their timestamp. However, this likely introduces some error because some blocks’ timestamps may be unreliable. A given block’s timestamp may even be earlier than the timestamp of a prior block. From Bitcoin wiki: “A timestamp is accepted as valid if it is greater than the median timestamp of previous 11 blocks, and less than the network-adjusted time + 2 hours.” For a more sophisticated approach, see the paper “Block arrivals in the Bitcoin blockchain” by R. Bowden, H.P. Keeler, A.E. Krzesinski, P.G. Taylor. [1]

Looking at data from 2016 [2], the mean block time is 9.61, and the distribution looks exponential, as expected. [3]

Now, if we only look at blocks that are found more than 10 minutes since the prior block, we see that the mean time is 19.64, about 10 minutes after our 10 minute offset. The distribution looks nearly identical, only it’s offset by 10.

The same pattern can be seen for offsets up to about 65 minutes. Beyond 65 minutes, although the expected time until the next block is still 10 minutes, we can’t show it empirically due to a small sample size.

We can take this one step further by picking a set of random times and observing the mean time until the next block.

Look familiar? It doesn’t matter what times you pick - 10 minutes since the last block was mined, totally random, every hour on the hour, your friends’ birthdays, whatever! The expected time until the next block will always be 10 minutes. [4]

[1] The paper takes a more sophisticated approach to identify blocks with unreliable timestamps, and then resamples them. They identified that 5% of blocks have unreliable timestamps.

[2] I chose 2016 because the BigQuery dataset has some missing blocks in 2017, which leads to outlier block times.

[3] The aforementioned paper Block arrivals in the Bitcoin blockchain rigorously investigates the assumption that Bitcoin blocks arrive according to a Poisson process, and hence time between blocks follow an exponential distribution. They note that Satoshi didn’t explicitly say that blocks arrive according to a Poisson process, but the original whitepaper includes calculations with this assumption: “Nakamoto [1] did not explicitly state that the blocks arrive according to a homogeneous Poisson process. However, the calculations in [1] are based on the assumption that the number of blocks that an attacker, mining at rate λ1, mines in the expected time z/λ2 that it takes for the community to mine z blocks at rate λ2, is Poisson with parameter λ1z/λ2, which is essentially equivalent to a Poisson process assumption.”

[4] For the same reason, the times since the most recent blocks were mined will, on average, be 10 minutes.

Full code:

OKRs

Wed, 02 Aug 2017 00:00:00 +0000

I’m a big fan of Objectives and Key Results (OKRs). Having worked both at companies that do and do not rely heavily on OKRs, I see the problems that occur when not using OKRs and the successes from using them.

Note: there are many resources on OKRs. My thinking is heavily informed by Rick Klau, Google, and Michael Dearing. For intro to OKRs, see here. This post is a collection of my most salient OKR thoughts.

Problems in organizations that OKRs help solve [1]:

Cross team visibility. What is Andy’s team working on? What about Joe’s? OKRs make answering these questions straightforward for anyone in the organization.
Expose duplicative efforts: Once you’ve clearly articulated what each team is working on, teams working on related things are exposed.
Prioritization: Writing OKRs is a forcing function: e.g. will we really work on the Slack integration?
Vertical alignment: Does an individual team’s work contribute to their broader team’s OKRs? And do subteams’ and individuals’ work contribute to their team’s OKRs?

Tips:

Revenue is generally not a good key result, because it is a trailing indicator.
OKR newcomers frequently write “launch product X” as either an objective or a key result. Both are bad. Your objective should answer “why are you launching the product in the first place?”, and your key results answer the question: “If we achieve our objective, what would be true?”
Key results are not todos, but rather indications that you’ve met your objective.
Rule of thumb: 3-5 objectives per team. This forces focus.
Rule of thumb: 3-5 key results per objective. Having 1 key result can lead to over optimization on one measure, and having too many can make measuring success difficult.

[1] Other goal setting frameworks exist and can be successful. But most other goal setting frameworks are really OKRs by another name.

"Home" by Molly James

Thu, 13 Oct 2016 00:00:00 +0000

I recently sat in with the talented Molly James at the Octopus Literary Salon in Oakland, CA. Starting at 3:15 I take a solo on a synthy organ sound.

Calculating Running Totals using SQL

Tue, 20 Oct 2015 00:00:00 +0000

Cross post from the Wagon blog. Thanks to the Wagon team for help with this blog post.

How many users joined in the last 5 months? What were total sales in Q2? How much revenue came from the March sign up cohort?

Although these questions can be answered with a single number, it can be useful to see a running total over time: how many unique users joined, or how much cumulative revenue was received by day over some period.

Usually, data is stored incrementally. For example, here’s a table of sales per day:

Date	Sales
10/1/2015	5
10/2/2015	3
10/3/2015	7
10/4/2015	8
10/5/2015	2
10/6/2015	3
10/7/2015	6

How do we generate the following table of cumulative sales over time? In SQL, there are two typical approaches: a self join or a window function.

Date	Running Total of Sales
10/1/2015	5
10/2/2015	8
10/3/2015	15
10/4/2015	23
10/5/2015	25
10/6/2015	28
10/7/2015	34

A self join is a query that compares a table to itself. In this case, we’re comparing each date to any date less than or equal to it in order to calculate the running total. Concretely, we take the sum of sales in the second table over every row that has a date less than or equal to the date coming from the first table. This is Postgres/Redshift syntax, but other SQL dialects are very similar.

This is not a bad approach; it is a nice showcase of how extensible SQL can be using only select, from, join, and group by statements.

But it is a lot of code for a simple task. Let’s try a window function. They are designed to calculate a metric over a set of rows. In our case, we want to sum every row where the date is less than or equal to the date in the current row.

The window function can filter and arrange the set of rows to run the function over. Here the order by date rows unbounded preceding limits the sum function to only sales before the date of the current row. Window functions are incredibly useful for time-based analytical queries; to learn more, the Postgres docs are a great place to start.

The final step of creating a chart and sharing it triumphantly with your teammates is easily accomplished using Wagon. Window functions for the win!

Calculating Active Users in SQL

Sun, 13 Sep 2015 00:00:00 +0000

Cross post from the Wagon blog. Thanks to the Wagon team for help with this blog post.

How engaged are your users? How frequently do they visit your website or app? Analytics services like Google Analytics and MixPanel calculate basic counts of daily, weekly, and monthly active users, but it’s difficult to customize or join these results with other data. Writing this query in SQL gives you more control. Let’s do it!

Here’s a table of user logins by day. How many users were active in the last week and month?

date	user_id	num_logins
10/1/15	1	3
10/1/15	2	null
10/1/15	3	1
10/2/15	1	null
10/2/15	2	1
10/2/15	3	3

Like calculating running totals, there are two approaches: self join or window function.

In either approach, it’s helpful to have a table of logins per user for each day, even if the user didn’t login (null in this example). If your data isn’t already organized like this, you can generate a table with a row per day, per user, with the following query (this is Postgres syntax, for other databases, modify the generate_series function to generate a table of dates).

To use this data, you can create a temporary table, use a common table expression, or include it as a subselect.

Approach 1: Self Join

A self join is when you join a table with itself. How meta is that? For each row, we ask how many logins that user had in the last week. The join condition requires emails to match and for the date to be within the last 7 days. In line 5, the query sums num_logins for those dates. The case statement identifies the user as active on that day if she had any logins in the prior week.

This query generates a table that tells us which users are seven-day-active over time. This result can be aggregated further, filtered for specific dates, used to find inactive users, and joined with other data. In Wagon, we can create a graph of the number of 7 day active users over time.

Approach 2: Window Functions

The self join works great, but modern databases have a more efficient way to get the same results. With window functions, we can explicitly aggregate only over rows that we care about with just a single pass through the data. If you have millions or billions of rows (lucky you), the self join will take a long time to compute. In line 5, the query sums num_logins for the user’s previous 14 days. It first partitions the table by email, then evaluates over a set of rows - in this case we’re looking at a specific date range. The case statement classifies the user as active or not just as before.

This query makes it easier to add additional metrics for 7 and 30 day active users. As expected, the wider your definition of active user, the more you’ll have. Use these new powers carefully!

Building an Analytics Pipeline in 2015

Thu, 06 Aug 2015 00:00:00 +0000

Cross post from the Wagon blog. Thanks to the Wagon team for help with this blog post.

Every company needs an analytics data platform queryable by SQL.

Using a single analytics tool or relying on logs from a single source is a fast way to get started but is rarely sufficient. You’ll realize you need a better data strategy when attempting more detailed analytics tasks: cohorting customers based on segments available in multiple data sources, analyzing long time scale trends, or making data available to other applications. Unfortunately, you’ll quickly reach the limit of your off-the-shelf tool.

There has been a dramatic increase in data being created and fascination with Big Data, but less of a corresponding uptick in how to capture its value. Engineering a system to ingest, transform, and process data from many (changing, flaky) sources has been a long time, Very Hard Problem™. Doing this well requires hard work – the dreaded ETL.

We see more and more companies choosing to invest in SQL warehouses and the requisite engineering well before they become large businesses. How do you effectively build one of these things? And what makes building robust analytics infrastructure difficult?

See full Google Trends report

Here’s an example illustrating the core problems: You implemented a new purchase flow in your app and you’d like to understand conversion rates (tracked from logs) broken down by your email marketing A/B test (tracked from a 3rd party). The log lines you’re generating have new structure and may need to be re-parsed to fit into your existing schema. The A/B testing info may live in a different place than user data. Boiler plate reporting tools and drag and drop analytics UIs are great, but they require structuring ahead of time and the new checkout flow change is already live in production. Manually doing this analysis one time is annoying, but turning it into a reliable, repeatable practice is nearly impossible without dedicated engineering effort.

Your goal should be to get your data into a data warehouse that can be queried directly by people and programs. While it’s not straightforward, it’s important to understand the pieces. We see companies addressing this problem by focusing on the following steps:

For each data source: generate, collect, and store your data
Transform data into usable, queryable form
Copy multiple sources into a single place
Enjoy the data fruits of your data labor

The first step is collecting the data with as much structure as possible. You need to generate the data, transmit it from apps, browsers, or services for collection, and then safely store it for later. Many mobile and web analytics providers offer these three steps, others focus on a subset. For example, Heap and Mixpanel generate many app usage events automatically. Others focus on receiving data and making it available to read later (Keen and Splunk as different examples). Segment takes advantage of the difficulty of logging to many places by transmitting data to many of the above services with one API call.

Another large source of data is logs (usually messy and unstructured). Just having logs is not enough - it must be massaged into usable rows and columns. Some log lines help engineers analyze technology performance or debug errors, some log lines must be interpreted to signal “human” events, and some log lines have been around for so long that no one remembers why they’re there. Logs are rarely generated with their end purpose or a fixed type system in mind. Transformation of these raw strings is necessary to make them usable rather than just searchable.

For example, you may need to combine three separate log lines in order to signal a successful-user-flow, or to compare log lines against prior data to understand if a user is new, active or re-activated. Or maybe you need to remove those extra pesky spaces around your beautiful integers or standardize timestamps across timezones. Trifacta, Paxata, and Tamr offer technical tools for transforming ugly log forms to structured rows and columns. Or you’ll roll your own.

Once data collection systems are in place, you want to get this data flowing into a data warehouse. While some of the aforementioned tools provide their own interface for accessing collected and processed data, joining across multiple sources is difficult if not impossible, and their interfaces are often inflexible and cumbersome. Luckily, many of these services recognize this, and offer easy exports to data warehouses. Amplitude and Segment do this well and offer straightforward exports to Redshift. Google Analytics offers export to BigQuery for Premium customers (~$150k / year). Others make it possible, but require a bit of work (for example, Keen). New startups like Textur and Alooma are working on plumbing data into hosted RDBMS systems.

Outside of dedicated analytics solutions, you often have data from third party sources you’d like to join and analyze (e.g. Salesforce, ZenDesk, MailChimp, etc.). Most of these tools offer APIs to extract data. Building and maintaining these connections from 3rd parties to your data warehouse on your own is doable, but this is often where data integration tools are helpful. Services like Snowplow and Fivetran help.

At this point, data is flowing in a structured way. But where is it going? When assessing a data warehouse, look for:

Large scale ingestion and storage
Fast computation and multi-user concurrency
Easy management and scaling with data
Security and permissioning

There are many that meet these criteria: Amazon Redshift, Google BigQuery, Microsoft Azure SQL Data Warehouse, Hive, Spark, Greenplum, Vertica, Impala (the list goes on!). The largest technology companies (Amazon, Google, Microsoft) are investing in, and subsidizing, these data warehousing solutions. It’s a crucial component to nearly every business, which naturally draws the attention of the tech titans.

It’s a data jungle out there. Diagram from 2014 by Matt Turck at FirstMark

Phew, now you can enjoy the freedom of querying structured data, and the work (and fun!) begins. We’ll have more on the data analysis step soon!

Wedding Speeches

Sun, 14 Jun 2015 00:00:00 +0000

I was recently given the great honor of playing a large role in the wedding ceremony of two good friends. I also gave a toast during the rehearsal dinner. I’ve copied both speeches below, as well as the best man speech I gave at my brother’s wedding one year ago.

Ceremony Speech

Welcome and congratulations to everyone who is here today, to witness and celebrate the uniting of Julia and Luke in marriage. I think we can all agree, that they have a beautiful and special love that has been cultivated and shared over the past six years.

As we stand here today, just a few miles from the majestic Appalachian Trail, I am reminded of Julia and Luke, as they are both avid, experienced hikers, rock climbers, runners, and world travelers. Yet, tonight, Julia and Luke, you are novices; (Pause) embarking on the first step of your most meaningful journey yet; your new life together.

Fortunately, you are not embarking on this voyage alone. You have the support of your loving parents, Annie and Robert, Nancy and Steve, siblings Rachel and David, your extended families, and all of your wonderful friends who have traveled long and far to celebrate this joyous occasion.

I’d like to take a bit of time to impart some wisdom that I’ve gathered from spending so much time with my dear friends Julia and Luke. Wisdom about hiking, about life, or perhaps about both. Here it goes:

Lesson number 1: Know your partner

Julia, Luke, you first met when you were RAs, resident advisors, of Few, your junior year at Duke. It was no coincidence that you were both drawn to and assumed this leadership position specifically geared toward helping others. Your young relationship grew through study sessions that often involved a bottle of wine, late night conversations that left you tired the next day, and rock climbing. Even then, you both pushed each other to new heights, literally. You enjoyed your first outdoor rock climbing trip together. It was on this trip that you established an ongoing pattern in your relationship: Luke, you continue to stretch Julia to her limits, and Julia, you reign Luke in from doing anything too dangerous.

Early on, Luke pushed Julia a bit too far while mountain biking in Duke Forrest. Julia tripped over a root and bloodied her knee. And much to Luke’s relief, Julia dusted herself right off and got back on the trail.

Lesson number 2: Maintain balance and have Fun

In order to maintain balance on the trail and also in life, you will need unique personality traits suited for different situations. Yes, you are opposites on the Meyers-Briggs personality type test! But you both know that this works in your favor. Life is always a balancing act, and together you do this so well.

For example, Luke, you are more reserved and you rely on your intuition. Julia knows what to expect from you: you’re consistent, loving, and stable. But never boring! You surprise her by being unpredictable, and Julia finds this quality delightful.

Julia, you are outgoing and spontaneous. You understand the world through sensing. You are a hard worker, serious, and ambitious. But sometimes just plain goofy. And Luke loves this about you!

Lesson number 3: Pack Enough Food

Julia is well known for her “hangriness”, a condition in which being hungry leads to being angry. Luke has gotten good at spotting this, and carries around his “hangry pouch” with some snacks and spare change just in case. It has saved the day on more than a few occasions. And this is just another example of how one takes care of the other.

Lesson number 3: Leave No Trace

Pick up all your garbage! But Julia and Luke, you both go one step farther than this classic outdoor mantra. Julia, you have an incredible social conscious, and Luke has a lot of respect for what you have accomplished in your life, from your work in global health in college, with developing countries at Google, and now with adoption at Binti.

Early on in your relationship, you bonded over reading Half the Sky, a book that is a call to arms against the oppression of women and girls in the developing world. You both share a deep concern for the world around you.

Julia, Luke told me that you have more compassion and empathy for others than anyone he has ever met. You care a lot about people and can deeply empathize with those from all backgrounds.

Lesson number 4: When the trail diverges, don’t be afraid to take the alternate route

Hiking for days on the AT might get monotonous. It will be boring at times. But remember that you are on an adventure!

Julia and Luke, you bring energy into everyday activities with each other and make them exciting. Luke, you told me that even going to the grocery store can be a thrill with Julia when you realize that it is an experience you get to have with someone you love.

When first greeting each other, you enjoy a big hug, say “Sweetie!” “Honey!” and do a celebratory dance. Every day you do this. Taking something mundane and elevating it into a meaningful moment.

Lesson number 5: Know Where You’re Headed

Perhaps most importantly, Julia, Luke, you share fundamental values and a common view on the world. You know the kind of life you want to live. You value experiences more than material possessions. Having come from such loving families, you recognize and place extreme importance on family. You respect each other. You appreciate the richness of life, and are able to savor it together.

In the words of Dr. Seuss, “You’re off to great places! Today is your day! Your mountain is waiting. So… get on your way.”

Julia, Luke, continue to enjoy each moment together, make every day an adventure, and spread the incredible amount of love you are feeling right now throughout the world. Congratulations.

Toast

Hello everyone! How are you all doing? What a nice dinner. And Asheville provides the perfect backdrop: A special place for a special weekend. Big congratulations to Julia, Luke, the Chou’s and the Chapin’s. Who’s excited for tomorrow? I know I am!

I first met Julia and Luke through my former roommate, Alec who worked with Luke at Accenture. Where’s Alec? Hi Alec! Knowing I was new to San Francisco, Julia and Luke welcomed me into their wide circle of friends right away. As you know, their organizational skills are legendary. They organize the much anticipated annual trip to Yosemite National Park, with about 20 amazing people, many of whom are here tonight. Hi Everybody! I know we all love to hike, but still, it’s nice sometimes to be within 20 feet of Julia and Luke, and to actually be sitting down. Resting. It’s strange. So I have Julia and Luke to thank for so many friends in San Francisco, and I know many of you here tonight can say the same.

As my friendship with Julia and Luke grew, they offered more than their ability to plan a really good hiking trip. They quickly began to offer me advice. Good suggestions, really. Perhaps unsolicited - but things I probably needed to hear.Get a new pair of glasses, Julia and Luke told me. Why? I only had them 10 years! But Julia, you were right. (turn to Julia) I needed a new pair. I did that, and updated my mid ‘90s look – which I still contend is a good look by the way - I was quite the fashion plate in 2nd grade. And now I trust them with all of my sartorial decisions. That’s why I, and my fellow groomsmen, have no trepidation at all about wearing the…. Salmon….colored… bow ties… that Luke has so graciously provided for us. Nope, no. Trepidation. At. All.

So, speaking of salmon, that reminds of the first time I met Mrs. Chou. Where is she? (look around the room for a while) Oh, there she is, hi Mrs. Chou! It might be time for another new pair of glasses. Anyway, I don’t know if you remember, but you were preparing a lovely dinner for us, and I had said that I like my salmon well done. Well, you must have taken me a bit too literally, as, next thing we knew, the salmon was ON FIRE. Luke was lightning fast with a water bottle to put it out, and as it happens, the salmon was not only unharmed, but delicious. You’re not catering the wedding dinner, are you? Just kidding! Mrs. Chou, like Julia, is a fantastic cook.

Joking aside, Julia and Luke are such generous friends - giving with their time and energy. Just a few weeks ago, Julia and Luke spent two hours with me thinking through a job decision. It takes special kind of people who are willing to spend so much of their time to help - let alone just a few weeks before their wedding day!

I feel grateful to have met Julia and Luke, and that I have been able to forge such a strong friendship with them. A big congratulations, and best of luck in the future. Let’s raise a glass to Julia, Luke, and a lifetime of health and happiness!

###Best Man Speech

Hello everyone! How are you all doing tonight? Wow! Is this a fantastic party or what?? Big thank you to Les and Denise. And how about that Hora? I think I may work that into my gym routine. [note to reader: the Hora at this wedding was 45 minutes long] For those who don’t know me, I’m Andy, Danny’s brother. Sorry to say - there’s only one of me [note to reader: my sister-in-law is one of five]. Mazel Tov to Jen and Danny, the Sterns, my parents and extended families. It’s been such a great day so far. And we are still going strong!

Any younger brothers out there? Well you may kick me out of the “Union of Younger Brothers” for saying this, but I feel very fortunate to have such a great brother like Danny. We have a ton in common, and love spending time with each other, especially on the golf course. And believe it ornot,until Jen entered Danny’s life, I was the source of one of his greatest pleasures in life. Beating me in a round of golf was truly pure joy for Danny.

(As an aside it doesn’t happen that often anymore, right Dan?)

But, really, none of that typical silly big brother stuff from Danny. No wedgies, noogies, wet willies, swirlies… no E’s in general. No, he was much more crafty than that. He made it SOO tough on me. I had to follow in his HUGE footsteps - How could I measure up to Danny? He was untouchable - academically, athletically, musically, comically - nobody makes me laugh as hard as Danny - even vertically and in every other way imaginable. And now he has set the BAR SO HIGH, matrimonially, by marrying the incomparable Jen that I may have to take up pole vaulting to get over this high bar.

I first met Jen when she came to our house for Shabbos dinner. I was immediately struck by how comfortable both Jen and Danny were with one another. And Jen’s level of comfort with Danny carried over to the rest of the family. It’s amazing how quickly and how well Jen fits in with our extended family.

But before Jen was a part of our family, there was an episode in the Jen and Danny courtship in which I played a small role, that I’d like to share.

I asked Danny how things were going with Jen, and he said, well, there is one thing, that Jen has been doing repeatedly, that the Granowitz family simply DOES. NOT. DO. So what was it? Jen eats popcorn……… at. the. movies! I said Danny, we’re the crazy ones here! Most people eat popcorn at the movies.

Now, a few weeks later Jen and Danny went to the movies with Grandma Ida. It turns out that Grandma Ida enjoys popcorn at the movies just as much as Jen! And this is just one small part of a special connection between them.

It doesn’t take great powers of observation to see how right Jen and Danny are for each other. The wonderful qualities that they share are there for all to see: solid family and religious values at the core. CHECK. Academic excellence and professional achievement. CHECK. Athletic and good looking. CHECK CHECK. Caring and fun loving leaders to their younger siblings. CHECK. Luckily, both Jen and Danny are smart and perceptive enough to have seen this as well.

Earlier in the week I was talking to Jen, and she thanked me for taking care of Danny before the wedding. I said no problem - thank you for taking care of Danny for the rest of your life!

I can’t begin to say how happy I am to have Jen join our family. I love spending time with Jen and Danny, I think largely because they love spending time with each other. Let’s raise a glass to Jen, Danny, and a lifetime of happiness! le Chaim!!

Special thank you to my parents who helped immensely with these speeches.

Querying CSVs in Wagon

Mon, 08 Jun 2015 00:00:00 +0000

Check out the Wagon blog for the full post.

Joining Wagon

Thu, 21 May 2015 00:00:00 +0000

Exciting news - I’m joining Wagon! I’m grateful for an incredible four years at Google, and thrilled to become a part of the incredible team at Wagon.

More to come!

IPython Notebooks

Thu, 30 Apr 2015 00:00:00 +0000

An IPython Notebook is one of the best ways I’ve seen to share data analysis.

On one end of the spectrum, you can share your code. Even if it is well commented, it can be difficult to really understand what is happening and replicate or modify it. On the other end of the spectrum, you can produce a write up. Even if it contains some code, graphs, and a clear description, it is not fully trustable without all of the code and data.

I've seen enough data analysis messed up in ways that flip the conclusion that I've stopped believing results without raw data and code
— Ben Hamner (@benhamner) April 5, 2015

Some folks produce a write up and then link to the code. This is good. Even better, from what I have seen, is an IPython notebook. It combines the best of both worlds: descriptive text explaining what is happening, all of the code and outputs, and an environment that enables you to reproduce the analysis.

Zeitgeist

Mon, 06 Apr 2015 00:00:00 +0000

Zeitgeist is defined as “the defining spirit or mood of a particular period of history as shown by the ideas and beliefs of the time.” Right now, there is an abundance of conversation among writers, academics, business people, politicians, etc., centered on the related themes of technology induced job automation and rising inequality. Related topics include STEM vs. the humanities, artificial intelligence, consciousness, and so on.

I may be falling prey to some cognitive biases that are causing me to overstate the “defining spirit” of our particular period in history (take your pick). But it does seem like the impact of technology on society is one of the dominant and recurring narratives among writers and thinkers today.

Here is a quick roundup of some things I’ve read recently on these topics. Not necessarily endorsements, simply articles that have come my way.

####Automation

The Robots Are Coming in the London Review of Books. A quote:

We are, Brynjolfsson and McAfee argue, on the verge of a new industrial revolution, one which will have as much impact on the world as the first one.

To me, this is the key question. Is the computer + internet “revolution” simply economic / technological progress that has very significant individual employment repercussions but ultimately no fundamental changes to our underlying economics? Or is it a fundamental change?

If it is a fundamental change on the scale of the industrial revolution, wow. From Wikipedia on the Industrial Revolution: “Economic historians are in agreement that the onset of the Industrial Revolution is the most important event in the history of humanity since the domestication of animals, plants and fire.”

####Inequality

NIMBYs in the twenty-first century in the Economist. A quote:

…a 26-year-old graduate student…presented a new paper…[that] several reputable economists regard…as the most serious and substantive critique that Mr Piketty’s work has yet faced.

Just the most recent in continued discussion over Piketty and issues of inequality.

####Humanities vs STEM

Fareed Zakaria’s column in the Washington Post on Why America’s obsession with STEM education is dangerous - he has a new book called In Defense of a Liberal Education.

Leon Wieseltier’s Among the Disrupted in the NY Times and much that was written in response. (not to mention the New Republic controversy)

####Artificial Intelligence

Of course much is written about incredible AI feats by companies and researchers, but perhaps more interesting is the growing number of prominent figures speaking out against its potential dangers. If you haven’t seen such threats, do a search for Bill Gates, Elon Musk, Stephen Hawking, Steve Wozniak, Sam Altman, etc. talking about the risks of artificial intelligence.

####Consciousness

I found this overview on research into the nature of consciousness fascinating.

Separately, I was reading an interesting but perhaps unnecessarily long article on Near Death Experiences and saw a related point:

…Rather, it shows that the mind and consciousness are emergent properties of the brain, knitted together somehow by all the physical and chemical processes in our nervous system.

But if so, then how does that knitting occur? This is the crucial question for consciousness studies…

…The question of how consciousness emerges is in fact likely to be one of the defining problems of the 21st century, when we might first be able to create machines as complex as the human brain.”

Perhaps more interesting than the link to AI is the connection to Yuval Harari’s closing remarks in his talk that I previously wrote about. Harari’s point is that we can find correlations between electrochemical patterns in the brain and subjective experiences, but we don’t truly understand the connection. Check out my other post for the full implications of this (which are drastic according to Harari, and relate to this zeitgeist).

Book Review: Spoiled Brats by Simon Rich

Sat, 28 Mar 2015 00:00:00 +0000

I first came across Simon Rich’s writing when his novella Sell Out was published in the New Yorker online in January 2013 [1]. Sell Out is about a turn of the (20th) century immigrant who gets stuck in a pickle jar and emerges in the modern day. I really enjoyed it, but only recently decided to give Spoiled Brats, a collection of short stories that includes Sell Out, a try.

From the book jacket summary, Spoiled Brats is a “collection of stories culled from the front lines of the millennial culture wars”. Most of the stories can be broadly described the same way: a light, humorous, and slightly critical commentary on millennial culture told through a story with supernatural elements.

For example, the first story, Animals, is similar to Sell Out in that it is commenting on today’s society from an outside perspective. In Sell Out, it is entirely clear where this perspective is coming from: a hardworking, Jewish immigrant from Eastern Europe who came to Ellis Island and worked in a pickle factory [2]. Yet in Animals, a story told from the perspective of an elementary school’s pet hamster, it is less clear what cultural norms the hamster is accustomed to, such that the hamster finds the school children deplorable (the only clue comes from this passage: “I, too, am a Christian, although lately I’ve struggled to make sense of God’s plan”). This makes the story a bit more shallow - which is okay given its short length.

Another common theme between the stories is how direct Rich is with social critiques. Here are a few excerpts from Sell Out:

“You could get interns.”

I raise my eyebrows; this word is unfamiliar.

“Interns?”

It takes long time, but eventually she is able to explain this thing to me.

“So they are slaves,” I say. “And it is not illegal.”

She hesitates.

“Basically.”

Another:

“Oh,” I say. “I forgot. You are wealthy.”

Claire turns around and stares at me. Her face is pale.

“Excuse me?”

“That is why you do not care about money. Because you already have so much of it. For you, all of life is happy game.”

Her eyes begin to twitch.

“That is so rude.” she says. “Life’s about more than money, Herschel!”

“Yes,” I say. “For those who already have it.”

Many of the stories contain similar tidbits.

Sell Out is definitely the centerpiece. It is the most engaging and funny. It’s not surprising to me that it is being turned into a movie. Yet the rest of the stories are also often laugh out loud funny, despite being slightly “syrupy and insubstantial” as a friend of mine noticed.

I’ll leave you with one quote that demonstrates how laugh out loud funny the stories can be. This one comes from Family Business:

I love my father, but sometimes he can get on my nerves. It’s hard to explain why exactly. It’s just the little things he does, here and there that bother me. For example, sometimes he shits into his hands and then throws the shit into my face while jumping up and down and screaming. I know he’s just trying to be funny - and it is funny, I can see that. But there’s just something about it that annoys me. I’ve asked him politely not to do it anymore, but I always get the same reaction. He just rolls his yellow eyes and says, “I’m sorry, your majesty.”

Footnotes:

[1] Part 1, Part 2, Part 3, and Part 4

[2] The historical accuracy of this perspective is highly questionable - for example, Herschel’s language (the narrator), the conditions and life he described, and so on - but this is mostly beside the point.

Talk Synopsis: Yuval Harari | Techno-Religions and Silicon Prophets

Tue, 24 Feb 2015 00:00:00 +0000

I recently enjoyed a YouTube video of a Yuval Harari talk titled “Techno-Religions and Silicon Prophets” - have a watch (perhaps sped up). Or if you’re pressed for time, read my synopsis below.

What follows is my interpretation of Harari’s central arguments. Below are Harari’s ideas and beliefs, not mine (necessarily). At times I quote from Harari directly and borrow his phrases generously.

–

Silicon Valley is where the most important religious ideas of today are originating [1]. The dominant ideology of today is liberal humanism. Here’s a definition of humanism from Google (liberalism, humanism, and liberal humanism are used interchangeably):

an outlook or system of thought attaching prime importance to human rather than divine or supernatural matters. Humanist beliefs stress the potential value and goodness of human beings, emphasize common human needs, and seek solely rational ways of solving human problems.

Here are some examples that demonstrate why liberal humanism is the dominant ideology of the day:

Voting: everyone decides for themselves and then a winner is chosen - “the voter knows best, no source or authority higher than the voter.”
Liberal economics: “the customer is always right”. A product is good if a customer buys it. There is no higher authority. Even if the smartest, wisest people produced what they deemed to be the best car, if nobody buys it, then it is considered a bad car. This is in contrast to communism, where a central unit controls the economy.
Love & marriage: the individual decides whom to marry based on his/her feelings.

Communism was the first “Techno-Religion” in that it promises a better life here instead of in the afterlife.

As demonstrated by many religious wars throughout history, as well as the Cold War, religion/ideology matters.

As non traditional religions such as liberal humanism and communism gained traction, traditional religions did not disappear but instead became less relevant. They moved from creative to reactive forces. For example, when contraception or the internet was invented, religious leaders needed to determine if it was okay and how to use it. And the reactive nature of traditional religions is not limited to technology. For example, traditional religions are still reacting to feminism.

Further cementing the idea that traditional religions have lost their footing as the dominant ideologies of the day, think about the most important discoveries of the 20th century. In the running: antibiotics, nuclear weapons, the computer, feminism, and so on. In contrast, it is difficult to think of any discovery on that level made by religions that believe in god.

For these reasons, China, radical Islam, and other older ideologies do not challenge liberalism. However, liberalism is not safe - because of what is happening in Silicon Valley.

The key insight to overtake liberalism comes from the life sciences. Liberalism believes in individualism. But according to the life sciences, humans have no soul, essence, or inner core. Instead, we are a collection of biochemical algorithms. And when you peel them away, there is no inner core. There is no individual.

The second key insight undermining liberalism is that these biochemical algorithms are not free. There is no freedom. Instead, they are either deterministic or random.

Taken together, these two ideas lead to the conclusion that an external system can understand humans perfectly, so long as the biochemical algorithms are adequately understood.

A few examples. Feelings (“the holy grail of liberalism”) are just biochemical algorithms calculating probabilities of survival and reproduction to inform our decision making.

Initially, this insight may seem to support liberalism in that it supports using your instinct and feelings - the byproduct of millions of years of evolution and natural selection - instead of the Bible.

But what happens if we create an algorithm that knows me better than I know myself? And this is the key point: when that happens, authority will shift away from me to the algorithms.

Right now, our feelings are still the best algorithms. But they are becoming outdated, and we are now in a position to produce an updated version. When we have better algorithms, the authority shifts to them.

To be clear, this has not happened yet, although it may be starting. Two examples come from medicine and health. Many of the most important decisions about our bodies are not taken by feelings but instead by external algorithms that understand our own internal systems and those of millions of other humans better than we do.

For example, Angelina Jolie had a preventative mastectomy because genetic testing told her that she had a high risk of getting breast cancer.

As this spreads to more and more fields, liberalism will collapse (not violently - it will simply become outdated). This has happened before with the transition from traditional religion to liberalism.

This is not just a matter of ideology, but a very practical matter of how to make decisions. Instead of consulting scripture or a religious leader, listen to your feelings. This represents the shift from religion to liberalism.

The next stage will be listen to a computer program. The supreme source of authority has already shifted from god to humans, and next will shift to data and information.

This may sound extreme, but consider this example on what might happen when you ask a computer whom to marry: (direct quote)

I know you from the moment you were born. I’ve read every email you’ve ever written. I’ve listened to every phone call. I remember every failed date. I can show you graphs of your sugar level and blood pressure for every date and sexual encounter in your life. I also know your potential mates, just as I know you. And based on all this information, and based on databases of millions of successful and unsuccessful relationships, I can recommend… with probability of 86% you go with [person] A instead of [person] B. I know you so well that I know you are disappointed, but you actually prefer B. You make this mistake because you give too much importance to physical appearance. Beauty is important, but you give it too much weight. Beauty counts for 9% of success of relationship, but your old fashioned biochemical algorithms, because of things that happened in African Savanna, give this beauty 27%, which is far too high. So even though you think B, go for A.

Another example might be Amazon’s book recommendations. Through the kindle, perhaps they will capture how quickly we read each book, our facial expressions, emotional reactions, etc., and use this data to become the authority on which books to read.

One might question whether the life sciences are correct about this. But it does not matter, because a religion or ideology does not need to be correct in order for it to spread (as evidenced by the creation story in the Bible, for example). Perhaps the new data algorithms will not be correct, but that will not prevent them from taking over the world.

–

One of the audience members asks Harari whether he thinks this is good or bad. Harari says in his book A Brief History of Humanity, you can read between the lines, and also wisely says, “If you start with a certain agenda, then very quickly you become blind to much of reality. one thing almost certain about reality is that it’s complex.”

The talk was interesting enough that I am curious to read his book or perhaps take his Coursera course on the same subject.

Fall 2016 Update: Harari’s piece in the New Yorker on the “Trump Moment” is worth a read. Harari argues that the “Trump Moment” signals liberalism’s end, and predicts “ideology that currently takes shape in Silicon Valley” will eventually reign.

Footnotes:

[1] Harari uses ideology and religion interchangeably. He states that sometimes religion is defined as belief in god(s), but that this is too narrow a definition. God plays a small role in religions like Buddhism and Confucianism. Harari defines religion/ideology based on the function it plays in society. A religion gives legitimacy to human laws and norms by hanging on to some superhuman law or power. There are two kinds of superhuman authorities: gods and natural laws.

Food Review: Beyond Meat (Veggie Chicken)

Sun, 01 Feb 2015 00:00:00 +0000

Although I am not a vegetarian, I have a proclivity for eating vegetarian cuisine. For vegetarian protein sources, I like nuts, beans, lentils, peas, high protein grains, seeds, etc. Including animal products, I enjoy eggs and greek yogurt. I tend to avoid fake meat products.

Yet after reading a profile of Beyond Meat in Outside Magazine, my curiosity (and tastebuds) piqued. Perhaps what interested me most were the testimonials from high profile customers, such as Bill Gates:

“I tasted Beyond Meat’s chicken alternative,” he [Bill Gates] wrote online, “and honestly couldn’t tell it from real chicken.” Gates quickly realized the blockbuster potential. “Our approach to food hasn’t changed much over the last 100 years. It’s ripe for reinvention. We’re just at the beginning of enormous innovation.”

Gates sat down with Brown [CEO of Beyond Meat] in 2012 and gave him some tips, which the entrepreneur took to heart. As Brown recalls, “He said to me, ‘If you get this thing to cost less than meat, and you get international quickly enough, then this is huge.’”

The entire article is worth a read if you are interested.

With such rave reviews, I headed to my local grocery and gave it a try. I bought the Grilled Chicken Strips and cooked them on the cast iron skillet as per instructions on the package. I mixed in some vegetables, and voila I had a vegetarian chicken stir fry (pictured below).

The verdict? Sad to say it failed to live up to the hype. It wasn’t bad tasting per se, but it was rubbery and relatively tasteless. It could not have passed for real chicken. Maybe I didn’t cook it correctly, but I don’t think so.

Given my inclination to avoid processed foods, I likely won’t try again.

However, Beyond Meat contends their products are not processed. Via Outside Magazine:

When I told Geistlinger [of Beyond Meat] that I was skeptical of processed foods, especially ones produced by novel techniques, he pointed out that Beyond Meat uses no artificial ingredients and employs the most time-tested of cooking methods (heat and pressure). “Our process is gentler than making pretzels,” he said. “Getting that browning on a pretzel requires chemically changing the bonds in the molecules. That’s more harsh than what we do.”

Food writer Mark Bittman disagrees. The article continues:

Mark Bittman, for one, is going to stay off the faux-meat bandwagon for now. “I think we have to evaluate each of these products individually,” he told me. “Some fake meats can easily pass for ‘real’ meat, but in many cases that’s because ‘real’ meat has been so degraded by the industrial production of animals. Still: the best direction for most of us is to eat unprocessed food of all types; fake meat hardly qualifies.”

Processed or not, healthy or unhealthy debates aside, I think my dish would have been more tasty had I substitued the fake meat for say, cashews or lentils.

I’m open to trying their forthcoming plant based hamburger (the “Beast Burger”), but I don’t foresee any fake meats entering my shopping cart on a regular basis anytime soon.

Twitter, Pocket, and Books

Thu, 15 Jan 2015 00:00:00 +0000

Twitter is my primary source of news discovery. And Pocket is my primary news consumption app.

When I have a few minutes to spare, I peruse Twitter for interesting articles that I want to read [1]. When I find one, I save it to Pocket. In addition to Twitter, I’ll save articles to Pocket that friends or coworkers email to me, that I find when browsing news websites directly, that are linked to from other articles, etc.

When I have more than a few minutes, I will open up Pocket and read articles I’ve saved.

This workflow separates content discovery and consumption, and is more productive for me than doing both at once. If I sit down to read articles in Pocket, I can get through many in a short period of time. No waiting for articles to load, no back and forth between apps. Focused, concentrated reading time.

When I first began using Pocket, I was pleased with the increase in reading output. I felt myself reading more “long form” and getting deeper into topics.

In fact, I’ve become such an active Pocket user that at the end of 2014, I received an email from Pocket informing me that I was in the top 1% of readers.

That’s right, I read 1.6 million words, which Pocket has calculated is about 35 books [2].

I was initially happy to see this. It confirmed my feeling that I was reading more using Pocket, which I thought of as a good thing. I still think so, but frankly I would rather have read 35 books! Or at least gone half and half.

A friend pointed out the irony that Pocket, despite being an app for reading digital content, still gives books the high merit of the benchmark. Can you imagine finishing a book and on the last page reading, “Awesome! You just read the equivalent of 500 news articles!”

Also ironically, Pocket’s congratulatory note may end up backfiring. It was a bit of a wakeup call - read more books, less news articles. There is a tradeoff, as I enjoy being up on current events. But books allow you to go even deeper, and provide continuity from one reading session to the next (among many other benefits, of course). In 2015, I may look to convert some of those 35 “books” I read into actual books.

Footnotes:

[1] Naturally, whom I follow on Twitter has a large impact on the news I am consuming. Therefore, I do spend some time curating the list of whom I follow. One of the reasons I like Twitter is because my feed is a unique blend of my various unrelated interests - something you cannot get from traditional media.

[2] That comes out to 46,694 words per book. On the short side, but otherwise seems like an okay assumption.

St. Petersburg Paradox Simulations

Mon, 12 Jan 2015 00:00:00 +0000

A friend recently posed a riddle to me:

You flip a fair coin. Tails, the game is over. Heads you win $2 and play again. In the next round, tails you take your $2 and the game is over. Heads you win an additional $4 and play again. In the next round, tails you take your $6 and the game is over. Heads you win an additional $8, and play again. And so on. The payoffs double in each round. How much would you be willing to pay to play this game?

If you are unfamiliar with this problem, it’s worthwhile to take some time to try and solve it. I needed pencil and paper [1]. When ready, scroll down to read on.

…

Now that you’ve had some time to think about it, take a moment to read through the solution in terms of expected value (this riddle is known as the St. Petersburg Paradox). I also enjoyed this Stanford Encyclopedia of Philosophy write up.

The key point is that the expected value is infinite:

So, in theory, you should be willing to pay a large amount of money to play this game (in particular, anything less than infinity). Clearly that is not a good idea - hence a paradox.

But what if you could play the game many times? By the law of large numbers, the more times you play, the closer the average of the results should be to the expected value (which in our case is infinite).

I was curious about what this would look like in reality, and wrote some R code to simulate playing the game many times. After running the code, I was initially surprised to see that even if you play the game a very large number of times, the average of the results was quite low, and quite variable. It was nowhere near approaching infinity.

The Stanford Encyclopedia of Philosophy includes a useful graph (reproduced below) that helps explain the interesting way the simulations of the St. Petersburg approach infinity.

Essentially, every once in a while, you’ll win big. This will cause a big jump in the average of the results thus far. The average will then slowly decrease (50% of the time you’ll win 0), until another big jump. Because the average of the results increases at each jump, it becomes more difficult to increase your average as time goes on. So the big jumps become more and more rare.

I ran this for 20,000 simulations:

Full code via this GitHub Gist:

[1] I find the more I struggle with a problem - math or otherwise - before looking at solutions, the deeper my understanding of the problem. This is the idea behind the aphorism "math is not a spectator sport".

Book Review: Think Like a Freak

Fri, 09 Jan 2015 00:00:00 +0000

I received Think Like a Freak, the third book in the Freakonomics series (following Freakonomics and Superfreakonomics) as a gift a while back, and finally got around to reading it.

As laid out by the title, this book differs a bit from Levitt and Dubner’s previous books in that it sets out to teach the reader how to “think like a freak”. Here’s an excerpt that explains what it means to think like a freak:

We’d like to bury the idea that there’s a right way and a wrong way, a smart way and a foolish way, a red way and a blue way. The modern world demands that we all think a bit more productively, more creatively, more rationally; that we think from a different angle, with a different set of muscles, with a different set of expectations; that we think with neither fear nor favor, with neither blind optimism nor sour skepticism. That we think like - ahem - a Freak.

Each chapter follows the same basic format. Through a series of stories (Levitt and Dubner later write that storytelling is a good method of persuasion), each chapter demonstrates a different technique employed by “Freaks”.

Although I found many of the broader messages on thinking like a freak not particularly memorable, I did enjoy many of the stories and examples told throughout the book. These stories were often anecdotes, pieces of research conducted by the authors or other academics, or mini experiments run by the authors.

One of the examples I found interesting was research on why Nigerian scam emails are seemingly so unbelievable, conducted by Cormac Herley, a computer scientist at Microsoft Research. In the words of Wikipedia, these scams are characterized by the “apparently comical, almost ludicrous nature of the promise of West African riches from a Nigerian Prince.” As outlined in Think Like a Freak, Herley explains that the absurd nature of these emails claiming to be from Nigerian princes is an advantage to the scammer, because only the most gullible of recipients replies. If the scammers began with a more believable email, they would receive way too many replies and would be unable to filter down to the people who would ultimately send the money. By making the email so outlandish, they are essentially inviting the most easily persuadable people to come forward and identify themselves.

One lesson I take away from this bit of research is that many seemingly stupid behaviors aren’t. But the broader approach Levitt and Dubner were trying to advocate was getting people to inadvertently admit something to you. Another example came from David Lee Roth, lead singer of Van Halen. He had lengthy technical and security specs that a hosting venue had to follow for all live performances. How did he ensure they were all followed? One requirement was no brown M&M’s backstage. Instead of inspecting the entire technical setup, Roth could simply look and see if there were brown M&M’s, and therefore learn whether all the detailed instructions had been read.

Think Like a Freak contains many such interesting stories and is a light and enjoyable read. However, one thing that left me curious was the continued glorification of Intellectual Ventures [1], a company that has been widely criticized for being a “patent troll” [2]. There is likely a portion of Intellectual Ventures that is indeed doing noteworthy and positive work. That the patent troll controversy is not mentioned in Think Like a Freak - not even in the extensive notes section - is a disappointing omission [3].

And this leads to a broader criticism about Think Like a Freak. Instead of a surface level discussion of how Intellectual Ventures handles failed projects, I would have preferred to hear Levitt and Dubner’s take on the firm’s roll as a “mass aggregator” of patents, and patent trolls as a whole.

More generally, in Think Like a Freak, Levitt and Dubner took a step away from unique perspectives on more substantial policy debates. Perhaps this is a response to the ultimately negative media attention generated from their position on global warming in Superfreakonomics (Levitt stands by what was written but admits that they “lost the media battle on the topic”). But to me it felt like they had less underlying source material to work with in Think Like a Freak than in the first two books.

As a high school student with no interest in economics, I was drawn to Freakonomics precisely because the off-beat topics interested me. I probably would not have picked up a book that discussed patent law, for example. Nonetheless, at this point in time, I would rather hear more substantive topics discussed.

In the final chapter of Think Like a Freak, the under-recognized merits of quitting are discussed. Levitt and Dubner ponder whether or not they will write a fourth book in the series, or “quit”. If they do, I’ll still likely read it, but it won’t be at the top of my reading list.

[1] Intellectual Ventures was also lauded in SuperFreakonomics.

[2] See Wikipedia entry on Intellectual Ventures

[3] Dubner was clearly aware of the allegations against Intellectual Ventures prior to the publication of Think Like a Freak.

On "In B Flat"

Fri, 09 Jan 2015 00:00:00 +0000

Try it out here http://www.inbflat.net/.

Things I like about In B Flat:

It allows you as the consumer of the experience to also be a part of the creation of the experience
Because of that, it can be different each time
Listening to it, you feel many different people coming together

And probably most of all, I enjoy the music that it produces. In B Flat is a tribute to Terry Riley, composer of In C (see In B Flat FAQs). In C is also a very interesting concept, but the music doesn’t make me feel the connections between the musicians in the same way as In B Flat does.

In B Flat is inspirational to me, in that I would love to see more opportunities for musicians from around the world collaboratively make music over the internet.

Storytelling

Thu, 08 Jan 2015 00:00:00 +0000

Me telling a story at a Spokespeople event.

For the past year or so, I’ve attended a few Spokespeople events. I’ve really enjoyed listening to other people’s stories, and learning about the art of storytelling.

A friend of mine has told many stories for Spokespeople. I asked him how he has so many stories to tell? He explained he thinks of his life as a series of stories. I agree - surely he has not had more story-worth events happen to him. He is just more capable of constructing the narrative.

I told my first story a few months back. Spokespeople liked it so much, they put it on their podcast!

Have a listen.

Improvisation

Mon, 01 Dec 2014 00:00:00 +0000

This post contains several completely improvised solo piano songs I recorded on GarageBand between February and April 2014.

In straight-ahead jazz (which likely includes a large portion of the jazz music a casual jazz fan would listen to), different composed songs are played [1]. Each song has a composed melody and harmony, and the rhythm and general feel of the song usually come from one or more of few different styles (e.g. swing, bossa nova, waltz, and so on). During some portion of the song, some of the instruments will "solo", whereby they improvise a new melody on top of the composed harmony and rhythm of the song. The improvised melody may be similar to the composed melody, or not, depending on the feel of the song and the style of the improviser.

This is in contrast to Free Jazz, where the composition may offer less melodic, harmonic, and rhythmic guidance (sometimes absolutely none at all). The improvised portion of the song forces the soloist to improvise all of these musical elements. Generally, Free Jazz is more atonal and less accessible to someone new to the music [2].

The recordings below differ from straight-ahead jazz in that there is no underlying composition that I am using as a base for my improvisation. I improvised the melody, harmony, and rhythm. I had no pre-conceived ideas what I would play before hitting the "record" button. However, they also differ from what would be considered Free Jazz in that it the songs are mostly standard in their use of harmony, and are mostly harmonious [3].

Many of the rhythms are similar throughout each different improvisation. I find improvising rhythms to be the most challenging of the three musical elements, especially while focusing on melody and harmony. The harmonies are improvised. I tend to draw on common harmonic patterns, but sometimes not. The melodies are almost always improvised (on occasion you may here a familiar sounding melody if it happened to pop into my head).

The biggest influence for these improvised pieces comes from Keith Jarrett. He is famous for performing totally improvised, solo piano concerts. His most famous is the Koln Concert (you should listen to it). I don't know much about how he approaches these improvisations, and I have never received any formal training on this style of improvisation.

Most of the other music I have posted here is older; in other words, a few years passed before I posted it. In that regard, that music passed a sort of test of time to my ears. I'm not sure how I will look back on this music in a few years. In general, I tend to like music I record shortly after I record it, but over time I may come to dislike it.

As of writing (May 2014), I mostly enjoy this music. It is totally unedited, so you will hear some mistakes. It is rough around the edges at times. Parts of it I already don't care for. But there are also segments that I really enjoy. Regardless of whether or not I enjoy the music afterwards, I greatly enjoy playing it. It can be deeply meditative. Oftentimes, when I stop playing, I will have no idea how long I had been playing for.

Have a listen below.

Footnotes:

[1] The compositions generally come from showtunes, classic swing or big band tunes, other popular music of the day, as well as compositions by the jazz musicians themselves.

[2] The seminal Free Jazz album is Ornette Coleman's The Shape of Jazz To Come. Recorded in 1959, relative to some of the Free Jazz that this album led to, it sounds rather tame and accessible.

[3] I say "mostly", because there are definitely some dissonant moments.

Is Tiger Winning

Fri, 07 Feb 2014 00:00:00 +0000

Watching the 2013 US Open with some casual golf fans, I realized they only wanted one question answered. With that in mind, I built www.IsTigerWinning.com. Much to my pleasant surprise, after a tweet and one submission to Reddit, I got over 10,000 Pageviews and 8,000 Unique Visitors in a little over 24 hours! And it even got picked up by Deadspin.

IsTigerWinning.com continues to update automatically. Although traffic declined steadily over the remainder of the 2013 season, traffic was highly correlated with how well Tiger was playing.

Here’s to you, time-pressed golf fan: www.IsTigerWinning.com.

Markov Music - a Simple Music Discovery App

Thu, 06 Jun 2013 00:00:00 +0000

I've always enjoyed music discovery services. When a new service is launched, I've been known to spend hours playing around seeing what new music the service can find for me. So I figured it would be fun to build my own music / artist discovery service (using Google App Engine). I call it Markov Music.

When you type something into the search bar, the app tries to match this search term to a known person. Using the Freebase database, the app then look for other artists that influenced and were influenced by the artist you entered. After showing you a YouTube video of this artist, another artist is randomly selected from the list of influenced and influenced by. And so forth and so on. Hence the name Markov Music - the artist selection technique forms a Markov Chain.

Give it a try here.

How to Play Golf on the Cheap

Wed, 06 Jun 2012 00:00:00 +0000

People frequently assert that in order to play golf, you must spend a lot of money. While it is certainly possible to spend a lot of money playing golf, you can be an avid golfer for a relatively small price tag. Let me explain.

Things I don’t spend any money on:

Golf clubs: I got my clubs over 8 years ago as a gift. So for the last 8 years I have spent $0 on new clubs.
Balls & tees: Between finding balls while playing (most rounds I finish net positive balls found vs lost), free or super inexpensive tees, and my long lasting golf bag, I have spent $0 on any such accessories in the past 8 years.
Golf cart: I enjoy golf much more when walking. If you’re going to be outside for 5 hours, you might as well get a little exercise. Plus, it saves what is usually a $20 fee to ride the golf cart.
Food: Golf course food tends to be decent, but nothing wrong with packing yourself a PB&J for lunch.

Things I do spend money on:

Greens fees: Hard to avoid this one. But great local courses, such as my home course Sterling Farms, cost under $30 for 18 holes.
Shoes: I buy one pair every few years, in the $50-100 range.
Gloves: A 3-pack from Costco is good for the season and costs about $20

And that’s it. The cost of shoes and gloves gets averaged out, meaning I pay about $30 for 5+ hours of quality, wholesome entertainment. Compare that to hour-long activities like going to the movies or bowling, and a round of golf starts to seem like a bargain.

Sterling Farms Golf Course (pictured above) costs $28 to play, and is a gorgeous, fun, and challenging golf course that I love playing.

Granted, golf can be very expensive. Some courses charge $500 per tee time. But there are lots of interesting, picturesque, and fun courses around in the $20-$40 range. I guarantee you can find one in your area.

When first taking up the sport, there are certainly some fixed costs, such as clubs, balls, and lessons. Here are some tips to help curb these initial expenses:

Buy a full knock-off set at Costco. You will get a bag, putter, driver, and full set of irons and wedges for $100-200. Alternatively, a used set that is a year or two old can be super cheap.
Ask a buddy for some old balls to use. Most longtime golfers (myself included) have literally hundreds of balls lying around. We’d be happy to hand some over.
Skip the golf shoes. Unless it’s raining, it really doesn’t matter - especially as a beginner. If you insist, save money by replacing the spikes when the shoes get worn out (instead of buying a new pair).
If physically able, always walk
Look for twilight rates - it’s a beautiful time of day to be outdoors, and you can get some big discounts
Investing in lessons can be valuable, but first I’d recommend checking out some of the many great resources available online. If you study up on your own first, you’ll get much better bang for your buck from a lesson or two with a good teacher.

Hopefully this post gives at least one person the impetus to take up golf. Or at least helps debunk the myth that golf is only for the wealthy.

Four Years Later, a Retrospective Review of GarageBand

Wed, 20 Jul 2011 00:00:00 +0000

As an amateur musician who loves to play, compose, and record music, I love GarageBand. I bought my Mac in 2007, so I am using GarageBand version 4 (the current version is 6). After a brief hiatus, I have started using GarageBand again, and decided to post what I’d like to see changed. If anyone is using a newer version, definitely let me know if GarageBand has improved in these areas. It’s also possible GarageBand has these features and I just don’t know about them! Again, please let me know.

Here are my thoughts:

Instrument Selection:

I own all of the Jam Packs (except for Vocals), and generally the selection of software instruments is phenomenal. I’ve spent hours playing around with the different sounds. They’ve done a great job creating realistic sounds that are fun to play. However, for all the variety, they have some stunning omissions. For example, while they offer many exotic instruments such as a Chinese Erhu Violin, Japanese Shakuhachi Flute, and even two varieties of the Turkish Lute, surprisingly absent are some common instruments such as a trumpet or a trombone. While they have great sounding orchestral trumpet or trombone sections, the individual instruments are absent.

Also, many common instruments that were included in the original GarageBand could use an update. The tenor sax, for example, is only marginally better than any old keyboard sound.

One possible solution to this problem is using MIDI instruments. I’ve used it for a trumpet sound - it beats the trumpet section when you’re looking for an individual trumpet. (Instructions how to do this here)
Individual Track Mixing:

Another great feature of GarageBand is that you can change the master volume throughout the song. This is useful for fade-ins and fade-outs, as well as inserting dynamics. However, you can’t adjust the volume of individual tracks throughout the song - only once for the entire song.

One way to work around this is to create another track of the same instrument, and divide your audio into chunks. This way, you can have two different volumes. It is okay, but not the best.
Tempo changes:

This, I believe, exists in newer versions of GarageBand. It is pretty frustrating to not be able to have tempo changes. No good ways to work around that.
Speed:

This is probably the biggest annoyance. It takes very long to load a song as well as an instrument. This probably has less to do with GarageBand and more to do with my 4 year old computer. Surely a newer machine and a newer GarageBand will be faster. In the meantime, the tips in the help article on optimizing GarageBand performance work pretty well.
MIDI Delay:

Again, this one doesn’t fall on GarageBand as much as it does on my hardware. I’ve got a very old “midiman” midi to USB. The delay from the time I hit a key to the time I hear it on GarageBand is small, but sometimes noticeable. For what I do, it isn’t too big a problem. But it would be nice if there was no delay.

A few other minor things: If it were up to me, I would improve the transcribing/notation features, add more metronome options, and keep making common instruments sound more realistic. The advertised guitar amp effects in the new GarageBand don’t appeal to me at all, although I’m sure some of the other less advertised features will. Please share any thoughts you may have, I’d love to hear from you.