Note: This my second blog post on Bitcoin/Blockchain. My previous postprovided an overview of Bitcoin/Blockchain and some brief commentary on the evolution of its applications, as well as its implications on securities trading. Of course, the potential impact is beyond just payment or exchanging non-physical assets. However, in order to truly appreciate the potential and shortcomings of Blockchain, one has to go back to the origin (Bitcoin and Blockchain were born together as one) and understand how the Bitcoin system works, ACTUALLY. Only then can one cut through the fog and begin to tell what is truth and what is myth, and start to appreciate the complexities. Inevitably, this Blog runs the risk of oversimplifying things and may appear naive to the true Bitcoin experts. But for the novice and non-techies, this is where we start diving into the rabbit hole…
Some relevant fun facts:
Figure A: Top ten crypto-currencies by market cap
Figure B: Bitcoin Blockchain Size = ~48GB (as of Nov 30th, 2015)
Section 0: Recap from Prologue
As we discussed in the last post, Bitcoin is not just a payment system or a currency. It is:
“A network-centric protocol and platform for recording [and transferring] ownership and trust on a peer-to-peer basis.”
The philosophical foundations of the Bitcoin system are:
- Completely open and decentralized
- Completely permission-less and borderless
- Un-censorable or uncontrollable
Some of these features are not original as P2P file/music sharing (such as Napster) ushered in openness and decentralization first. But in the world of finance, they are nothing short of radical and disruptive. The key building blocks and concepts of the Bitcoin systems are below. Though imperfect, I have provided examples from our daily lives to facilitate understanding by analogy. We will explore these concepts in a little more detail later and hopefully in layman’s language:
- Creating Bitcoin transactions vs. Swiping credit cards
- Creating and validating a block of transactions vs. Batch processing of transactions
- Mining (Proof-of-work) vs. Not necessary in a centralized or permissioned environment
- Blockchain vs. A Database
- Bitcoin incentives vs. Credit card interchange fees
Before we explore how a Bitcoin transaction works, let’s first explore the differences between a centralized and a decentralized system, using payment as an example. In this case, the Bitcoin payment system vs. the credit card payment system (Visa and MasterCard). This comparison will help us understand at a high level the philosophical and mechanical foundation upon which Bitcoin was created, as well as providing a tangible context to understand the role of Blockchain. So let’s begin…
Section 1: Bitcoin System vs. Credit Card System
1.1 Credit Card System:
Figure 1: Below is a simplified topography of the credit card payment systems in the U.S.
The card networks were born in the 1960s and matured in the 1970s. They began as bank owned associations until they become independent public companies in 2006 (MasterCard) and 2008 (Visa). The establishment of centralized credit card networks is instrumental to the creation of the highly profitable credit card business for the banks. In a way, the card networks were able to grow because the banks “allowed” it to. This payment system (an open loop system) operates on ahub-and-spoke model and relies on trusted intermediaries – card networks and banks – to connect end parties to transact (e.g. you the consumer and your neighborhood Starbucks), manage routing of transactions, process and clear transactions through dedicated networks. In this model, trust is a “pre-existing property” when a reliable central authority sits in the middle.
1.2 Bitcoin System:
Figure 2: Below is the topography of the Bitcoin system, born in 2008 and still evolving.
Unlike traditional banking and payment systems, there is no trusted central authority. Bitcoin is structured based on a P2P architecture on top of the internet. P2P means that nodes/participants – users and their machines, be it a smart phone, laptop, desktop, or dedicated servers – all have equal rights to verify and process transactions, to write and read from the Blockchain. In the Bitcoin system, trust is an “emergent property” from the interactions of different participants, which is governed by the Bitcoin protocol (this concept may be hard to grasp, but we will circle back to this as we explore how a transaction actually works).
Key Takeway: The philosophical and architectural difference between the two systems could not be more far apart.
1.3 Card Networks vs. Bitcoin Nodes – Their Roles and Value Add
According to Carol Coye Benson and Scott Loftesness, authors of Payment System Systems in the U.S. Card networks provide several key functions that are essential to the functioning the credit card payment system:
- Processing – this means transaction routing and settlement, the way in which transactions moves from one bank (or its processor) to another for payment authorization, and how gazillions of payments are netted and settled between the banks.
- Setting rules – this means setting and regulating the operating and economic rules that bind the behavior of intermediaries, enabling inter-operability and efficient dispute resolution.
- Managing the payment brand – brand advertising that educates the public about the means of payment, which has attracted consumers and merchants from cash to credit card as a trusted payment method.
Given the above functions, card networks provide the benefits of a standardized payment product experience, economies of scale in transaction routing, and a global payment framework that no one bank could develop on its own. The interoperability enables a customer from China walking into a store on Fifth Avenue to buy a Prada handbag without any hassle. This is the value add of card networks.
While membership in the card network is strictly limited to banks and other regulated financial institutions, participation on the Bitcoin network is open to anyone (permission-less). In fact, I downloaded and installed a Bitcoin client from Bitcoin.org and downloaded a full-copy of the Blockchain on my desktop. Boom, I have become a node on the Bitcoin network!
Figure 3: The look of desktop Bitcoin client
A node could perform one or all four of the following functions:
- Creating transactions – in Bitcoin term this means serving as “a Wallet”
- Verifying and settling transactions – in Bitcoin term this means “Mining”
- Routing transactions – in Bitcoin term this means propagating a transaction or a block of transactions to other nodes on the P2P Network
- Storing transaction data – in Bitcoin term this means maintaining a sub-set or a full copy of the Blockchain on the hard drive
Theoretically, any piece of everyday computing hardware could serve all four functions. But practically the desktop that I customized four years ago could not perform the “Mining” function as there is not enough computing power (nor is there incentive for me, as you will see later), and my iPhone could serve as “a Wallet” but not store a full-copy of the Blockchain due to the limited size of its flash memory.
Key Takeaway: At the risk of over simplification, in the credit card ecosystem, customers initiate transactions, banks extend credit and maintain accounts, and the card networks do the plumbing. In the Bitcoin system, the tasks performed by participants are determined by their objectives and incentives (to make a payment or earn fees through mining) and the type of activity their hardware could support based on CPU power and storage etc. The essential plumbing job performed by the likes of Visa and MasterCard has been shifted to individual nodes/participants on the Bitcoin network,
Now that we have established the high level differences between the two systems. Let’s trace the life cycle of a Bitcoin transaction to understand how it actually works.
Section 2: Credit Card Transaction vs. Bitcoin Transactions
Regardless of the type of system, a transaction’s life cycle starts with its creation, it is then processed (approved or declined), settled, and lastly logged into a database for recordkeeping or reporting. Let’s see how transactions work on the Credit Card Network vs. the Bitcoin Network
2.1 Credit Card Transaction Life Cycle
The figures below offer a clear illustration of a transaction life cycle on a credit card network, which consists of two processes – an authorization processand a settlement process. For a more detailed explanation, I recommend reading Payment Systems in the U.S. by Carol Coye Benson and Scott Loftesness.
Figure 4: Credit Card Transaction Authorization Process
This is what happens at the point of sale. When you swipe your credit card at a Starbucks for a $3 cup of latte, the credit card terminal (or could be a Square card swiper) connects to the credit card network and your bank to check if your account is valid and if there are sufficient funds to cover the transaction. At this step, the $3 are held and deducted from your credit limit but not yet transferred to Starbucks’ account. This process is instantaneous to ensure a frictionless payment experience.
Figure 5: Clearing and Settlement Process
This is what happens at the end of the business day. All the transactions occurred at Starbucks are “batched together” and submitted through the credit card networks for settlement, where funds are transferred from your bank account to Starbucks’ bank account. Settlement looks like a B2C process in this example, but it is also a B2B process. Your bank (BofA) and Starbucks’ bank (Chase) probably serve thousands other merchants and customers. During a business day, they are instructed by thousands of account holders to pay each other. At the end of the day, these debits and credits are netted, if net-net BofA owes Chase, instead of transferring a large number of small payments to each other (which is also possible), BofA just makes one payment to Chase. Though the real world is more complicated, this is the essence of transaction settlement. Contrary to common sense, settlement is not instantaneous and could take up to three days for the $3 to be deposited in Starbucks’ account.
Here I would like to revisit the concept of “Pre-existing Trust,” because banks and card networks stand in between consumers and merchants to take on the credit and fraud risk to ensure that the counter-parties could confidently deal with each other, that merchants get paid, trust is pre-existing in the system, and lies with the financial institutions. The system is imperfect, but it works almost without friction, and the intermediaries charge handsomely for providing this trust and convenience (2-3% of transaction value and sometimes fixed fees per transaction).
2.2 Bitcoin Transactions – Overview:
Now let’s walk through a bitcoin transaction. According to Bitcoin inventor Satoshi Nakamoto’s original white paper, below are the steps in a transaction life cycle on the Bitcoin network:
- “New transactions are created and broadcasted to all nodes”
- “Each node collects new transactions into a block”
- “Each node works on finding a difficult proof-of-work for its block”
- “When a node finds a proof-of-work, it broadcasts the block to all nodes”
- “Nodes accept the block only if all transactions in it are valid and not already spent”
- “Nodes express their acceptance of the block by working on creating the next block in the chain, using the hash of the accepted block as the previous hash”
There are some key terms here such as a block, proof-of-work, and Blockchain, we will discuss them as we navigate through each steps.
Section 3: Creating Bitcoin Transactions
3.1 Step 1 – New transactions are created and broadcasted to all nodes
1. Anatomy of a Bitcoin transaction
While a credit card references an account as the funding source, a Bitcoin transaction references a specific previous transaction as the funding source (inputs). With a $5000 unspent credit on a credit card, there is no difference between the first $200 and the rest $4800 in that they all belong to a general pool. In the Bitcoin system, each unit of Bitcoin has its own digital identity. This is accomplished by associating a Bitcoin either as an input or output of a transaction and attaching it with a Bitcoin address that identifies an owner.
Again, a Bitcoin transaction references a specific previous transaction as the funding source (inputs). This means one transaction’s output becomes another transaction’s input and so on, thus creating a chain of ownership as the Bitcoin is moved from one Bitcoin address to another. In essence, all the transactions are “linked,” providing the digital footprint to ensure that a unit of Bitcoin has not been double-spent, meaning become the input for multiple transactions. Spending in credit card is like draining a pool of money, “Spending” in Bitcoin is like picking coins out of a jar, in this case gathering unspent Bitcoins from multiple previous transactions to fund the next transaction, transferring value to a new owner identified by his or her Bitcoin address.
Figure 6: Transactions in the Bitcoin system, where the output of one transaction is the input for the next transaction
Source: Mastering Bitcoin: Unlocking Digital Cryptocurrency by Andreas M. Antonopoulos
Notice in the figure above, the transaction input does not add up to the output to the recipient’s Bitcoin address, the difference is a transaction fee earmarked for the “miner” who successfully create the next block of transactions . This is one of two incentives for “miners” to process transactions. The other newly issued Bitcoins when a new Block of transactions is created, which we will discuss later.
All historical transactions are recorded on the Blockchain (a ledger) and completely transparent. Transaction linking and total transparency serve as the foundation that enables each participant on the network to independently verify if a new transaction is valid.
2. Constructing a Bitcoin Transaction
To initiate a new transaction, a user would need to use a Bitcoin client that could perform the “Wallet” and routing functions. For instance, I use the “Wallet” application Coinbase.
Figure 7: My actual Bitcoin wallet
Wallets do not actually hold Bitcoins as they are inputs and outputs registered on the Blockchain. What Wallets hold are keys. Here we introduce a few new concepts:
- Private key – think of this as the secret Password to your bank account orSignature on a check. Private key is a just random number and is used to generate a public key.
- Public Key – think of this as the Account Number of your bank account. A public key is calculated from the private key using a one-way mathematical function, meaning it cannot be solved backward.
- Bitcoin Address – think of this as Beneficiary Name on a check. Bitcoin address is used to represent a public key and is a string of numbers of letters generated from it, again using a one-way mathematical function. Bitcoin address is what you could share with anyone who want to send money to you.
Figure 8: Relationship between private key, public key, and Bitcoin address
Source: Mastering Bitcoin: Unlocking Digital Cryptocurrency by Andreas M. Antonopoulos
Keys and addresses are important in that they establish ownership of Bitcoins and enable many key features of the Bitcoin system – namely decentralized trust and control. Bitcoin transactions require valid signatures from the recipient to be included in the Blockchain (as part of the verification process). Private key generates this signature to prove that the recipient is the valid owner of the Bitcoin address and the Bitcoin address in turn associates ownership of the Bitcoins to the recipient. Private key is used to control access to all unspent Bitcoin (from previous transactions) associated with the corresponding Bitcoin addresses. The one-way functions ensure that no one can gain access to your private key using your Bitcoin address or public key.
So Bitcoin Wallets contain keys not Bitcoins. The Bitcoins are stored on the Blockchain in the form of unspent transaction outputs. To initiate a new payment transaction, my Bitcoin Wallet App would:
- Queries through a pile of unspent transaction outputs for my Bitcoin addresses or addresses on the Blockchain until it could find enough money to fund (use as inputs) the new transaction.
- The Wallet app constructs a transaction output (a payment) to Starbucks’s Bitcoin address, which Starbuck could redeem by presenting a signature from the private key corresponding to its Bitcoin address, thus unlocking the output as an input for the next transaction e.g. paying a coffee bean supplier.
- Once the transaction is constructed, my Wallet App transmits it to the Bitcoin network for it to be verified by other participants on the network and eventually added to the Blockchain
3. Broadcasting a Bitcoin transaction
Here is how a Bitcoin transaction message travels through the network. First, my wallet app sends the new transaction message to other nodes it is connected to over the internet. Any node that has not seen this new transaction will independently check it against a checklist of criteria before forwarding it to other nodes to which they are connected (e.g. the referenced output has not been spent, the construct of the transaction message is in compliance with the Bitcoin protocol). Only transactions confirming to the criteria will be forwarded, while invalid transactions will be discarded by the node that received it. In a matter of a few seconds, the verified (but unconfirmed) transaction propagates across the Bitcoin network, reaching a large number of nodes. To visualize this process, imagine fast forwarding the growth of a tree.
To draw a comparison against the centralized Credit Card system, transaction messages all travel through Visa and MasterCard and get routed to the intended end party. Imagine this as a toll station on a highway system.
Section 4: Bitcoin Mining and Emergent Consensus/Trust
4.1 Summary for Step 2-6
In this part of the blog, it is helpful to talk about step 2 – 6 together at a high level as they represent the transaction processing and trust building process in the Bitcoin system – “Mining”. In short, mining is essentially a giant computational race among all the mining nodes to be the first to collect a pool of valid transactions, construct a new candidate block, solve a mathematical puzzle, and create the next valid block accepted by a critical mass of participants to be added to Blockchain. Mining nodes are specialized computing hardware set up for the purpose of performing the transaction processing function and solving the mathematical puzzle called proof-of-work. These nodes store a full-copy of the Blockchain.
According to the Bitcoin protocol, the difficulty of the puzzle is set and self-adjusting so that on average a new block is created every 10 minutes across the entire Bitcoin network. This 10-minute interval sets the tempo for the entire Bitcoin network and determines the speed at which a transaction is verified and settled. Every 10 minutes, a winner is announced (a new block is created) and the game is reset, and all the nodes get back to work to create the next block. Who could find the solution to the puzzle and get to create the next block is purely probabilistic depending on the miner’s computing power. Thus in a decentralized network, barring any dishonest person or a group of people controlling 51% of the computing power across the network, whoever wins the race is by and large random. This ensures that no one entity could take control of the Blockchain by winning the race most of the time, which enables rewriting transaction history to allow double-spending, or cornering the market for transactions fees and newly minted Bitcoins. We will talk about transaction fees and Bitcoin minting when we discuss incentives built into the Bitcoin system.
The logic above seems abstract at first glance, but at its core, it is remarkably simple. Imagine a race among a large number of people competing to reach a series of destinations on an infinite track. According to rules of the game, every time a person wants to rewrite history, he/she has to go back to a previous destination and starts running again to catch up to the other people. If he/she runs really fast (has a lot of computing power), it might be possible, but the further back he/she goes, the harder it is to catch up with everyone. And among a large group of honest and fast runners, the chance is quite slim that he/she will be able to do this every time. Thus the bad guys have no incentive to try because it is practically impossible.
Let’s tie it together now. What is trust in the Bitcoin system then? It is the confidence that the longest Blockchain is the golden copy of truth, it is the confidence that all the unspent transaction outputs logged on that Blockchain is accurate, it is the confidence that when I pay Starbucks $3 for that cup of latte, I indeed own that $3 and it has not been spent somewhere else. The more time passes, the harder it is for people to change older parts of that golden copy of truth, and the more trust there is in the system. Boom!
In summary, Bitcoin mining serves two purposes:
- Validating transactions against the current blockchain, processing new transactions to add to the Blockchain
- In the process, enabling trust to emerge through consensus on whether these processed transactions are truly valid (unspent).
Having explained the high level concept of mining. Let’s explore how it actually works:
4.2 Step 2 – Each node collects new transactions into a block
Assuming a new block has just been added to the Blockchain, the game starts afresh and the miners race to complete the following steps to create the next Block:
- Collecting new transactions – as new transactions are constantly entering the Bitcoin network from Wallet Apps and other Bitcoin clients, including my hypothetical payment to Starbucks, mining nodes start adding them to a temporary pool of unverified transactions maintained by each node.
- Independently verifying new transactions and constructing candidate block – After independently verifying the transactions against the aforementioned check-list, the node will propagate a pool of valid (but unconfirmed) transactions to other nodes and start aggregating and constructing them into the candidate block (a block typically holds a few hundred transactions and has a maximum size limit in kilobytes). The selection of which transaction to include in the block first is based on a prioritization metric, which is a combination of transaction fees and a few other criteria. We will talk about transaction fees when we discuss the incentives in the Bitcoin system.
- Adding the Generation Transaction – actually before adding other transactions to the candidate block, a special transaction call Generation Transaction or Coinbase Transaction is added. This transaction does not have a previous transaction as a reference but is created by the mining node, according to the Bitcoin protocol, as a reward payment to the miner’s Bitcoin address for doing the mining work. So this is thesecond type of incentive for miners and how new supply of Bitcoin is created. According to the Bitcoin protocal, the reward for each new block will decrease every four year and currently stands at 25 Bitcoin per Block. At today’s price per Bitcoin of ~$370, that is ~$9250 per Block. But to earn this reward and all the fees associated with transactions included in the block, the miner must win the computational race to solve the Proof-of-Work.
- Constructing Block Header – to finish constructing the candidate Block the mining node will also need to construct the Block Header by filling in six fields as listed below. The Block Header serves the following purposes:
- First, consider it as the digital fingerprint or summary for a block of transactions. This tells which previous block (parent block) this new block is supposed to be linked to, using the Previous Block Hash field. This is important for assembling blocks into chains.
- It is the input for and output of the Proof-of-work Algorithm, which is crucial in making the block valid and for enabling the decentralized consensus mechanism to converge on the golden copy of truth (remember there are many instances of the Blockchain out there). We will discuss Proof-of-Work in the next section.
Figure 9: Structure of a Block Header, which has six data fields that need to be filled.
Out of the six fields, four are given (listed below), one (The Nonce) is a variable for the time being, and the other one (Merkle Root) needs to be filled using a one-way hash function.
- Version number
- Previous Block Hash: output of the Proof-of-Work for the previous Block of transactions
- Difficulty Target
The Nonce is a variable subject to change for the Proof-of-Work, discussed next.
The Merkle Root is found by using one-way cryptographic hash functions again (remember one-way hash functions are used to generate Bitcoin address from keys). This is done by feeding all transaction data through a one-way hash function, concatenating the hash value of two transactions at a time, then feeding the concatenated hash value through another hash function. This is repeated until all transactions are summarized by a single hash value. This process forms a tree like formation called the Merkle Tree. The one single hash value left is called the Merkle Root.
Figure 10: Fill in the Merkle Root field of the Block Header
One-way cryptographic hash functions (different types) are used extensively throughout the Bitcoin system:
- Generating Bitcoin Address
- Creating the Merkle Root
- Proving Proof-of-Work
On the one hand, they form the foundation for the security model of the Bitcoin network. On the other hand, it efficiently summarizes and protects the integrity of large data sets as they turn data of any size into data of fixed size, thus offering the potential to save storage space. A full discussion of hash functions are beyond this blog.
4.3 Step 3 – Each node works on finding a difficult proof-of-work for its block
With all fields of the Block Header filled and the nonce field is left at an initial value of 0, a candidate block has been constructed, but it is not valid until the mining node finds a solution to the Proof-of-Work algorithm. This process of finding a solution to the algorithm is known as “Mining”.
In short, mining is the process of changing one field in the Block header, in this case the nonce, and feeding the block header through the hash function (aka. hashing the block header) repeatedly until the resulting hash value is less than the difficulty target.
It is not possible to know the solution in advance and it can only be found through trial and error. Thus the speed of finding a solution is purely determined by the computing power of the miner’s machine, meaning how many hashes (billions…) it could perform per second. The mining process comes at the cost of electricity to keep the computers running, and the reward is earning the transaction fee and newly minted Bitcoin if the miner finds the solution faster than everyone else. In mining, it is a winner takes all situation. We will discuss the creation of new Bitcoin when we discuss the incentives. Once the mining node finds the right solution, the Block is valid and can be broadcasted to other nodes on the network for validation.
4.4 Step 4 – When a node finds a proof-of-work, it broadcasts the block to all nodes
Once the mining node found the solution, it immediately sends the Block to peer nodes it is connected to, and the Block is validated and propagated in the same fashion as how new transactions travel through the Bitcoin network, as we discussed before.
4.5 Step 5 – Nodes accept the block only if all transactions in it are valid and not already spent
Once nodes receive and validate the Block against a check-list of criteria (e.g. no transactions have been double spent, and the solution to the Proof-of-Work Algorithm is correct), they abandon their efforts to create a block at the same height and immediately start trying to create the next Block in the chain.
4.6 Step 6 – Nodes express their acceptance of the block by working on creating the next block in the chain, using the hash of the accepted block as the previous hash
Nothing much to explain about the above step at this point, but you may wonder how the different nodes come to a consensus on which Blockchain is the golden copy. After all, there are different copies of the Blockchain stored on many different full mining nodes and each node is constantly trying to create new Blocks to add to it. How do conflicts and inconsistencies get resolved in this completely decentralized network. Yes, inconsistencies do happen from time to time. These temporary differences are called Blockchain forks. These forks are typically resolved within one block along the Blockchain. To resolve the differences:
The mining nodes stand ready to abandon their work-in-progress and always choose to build on the Blockchain that has the most cumulative difficulty level (as recorded in each Block Header) that represents the most proof-of-work undertaken.
Typically, this is also the longest Blockchain in terms of number of Blocks in it. As long as all nodes follow the above rule, the entire Bitcoin network eventually converges toward the same copy. The following simplified example illustrates how this takes place:
Figure 11: Before a fork take place, the global Bitcoin network agrees on that Block P is the last Block in the chain.
Figure 12: Two miners from different parts of the world found two blocks (red and green) almost simultaneously and both reference Block P (blue) as the previous Block.
Figure 13: Now the two different blocks (Red and Green) are propagated across different parts of the Network, some nodes receive Block A first and chain it to Block P, other nodes receive Block B first and will ignore Block A for the moment. Thus the network is split in two with regard to which Blockchain is the golden copy, notwithstanding the inconsistency, each part of the network will race ahead to create the next block.
Figure 14: Now the group working on the Green fork of the Blockchain just found another Block that extends it, thus creating the longest Blockchain. The entire network at this point will re-converge to the Blockchian with the Pink block as the golden copy.
Section 5: Wrap-up
Thus this concludes my humble attempt at explaining how Bitcoin/Blockchain actually works. The key concepts and pillars of the Bitcoin system are:
- The system is completely decentralized, transparent, un-censorable, and borderless
- Each transaction references one or more previous transactions as its input (funding source), thus creating a traceable chain of ownership
- Each block of transactions are linked to a previous block, laying the race track for miners
- Miners serve the critical function of independently verifying and validating transactions, as well as blocks of transactions
- Trust is not a pre-existing property in the system, but an emergent property through interactions of participants on the network, and Proof-of-Work is a key building block of this interaction.
Whoever came up with the Bitcoin system is simply genius…the concepts and the mechanics of the system are simply two or three degrees ahead of our time. A piece of coin is the most physical incarnation of money/value, and Bitcoin is probably the most abstract form of it, but somehow trust is formed in this completely decentralized environment, governed by the invisible hand of the Bitcoin protocol, and we are just at the very beginning of exploring its full potential and use cases. “Stay hungry, stay foolish”