一还在做学生的时候，跟朋友开车从达拉斯到另一个city去吃饭，走的是人称鬼见愁的I65公路，因为基本上都是顶着限速或者无视限速在开。当时还是白天下午，回达拉斯的路上，车流在拐弯处减速，汇聚然后产生了堵车跟车。当时因为其实路上车不算很多，所以肉眼可见的跟车，我就提前减速了下来。我自认开车还算比较稳当的，因为我是一个谨慎胆小的人。然后就是dong的一声，我和坐在副驾的朋友以及我的玻璃水同时从我们在的地方飞了起来。我心里想的是完了我的脆皮日系车估计要完。下车一看雅阁威武，只有对方车子车牌螺丝的两个螺丝印。 虽然是第一次遇到交通事故但是已经网上看过很生活经验贴，也在脑子里演练过，但是接下来发生的事情仍然是超出了想象。车上下来一个满身味道的大姐眼神迷离，我顿感无奈，连忙让朋友去拍他的车型和车牌号了，我说那你给我你的保险吧。一般来说，买全险就是出了事故保险公司出钱修两方，半险liability就是修对方。搞笑的是，我们当时还在高速上，路过的朋友们在驰骋的过程中仍不忘摇下车窗来一句"Congratulations!"。对面姐们支支吾吾，一会给一会不给。我同时开始报警，“有人受伤吗？”“没有”，“有人需要救护车吗”， “应该没有”，“那请你们交换保险驾照信息然后尽快驶离”。然后姐们估计不想警察来，我电话还没挂，匆忙之间让我拍了她的保险就上车走了。我们也就无语的开车走了。二我后续做了三件事：第一件事 ...继续阅读 (8)

上周末出门跟着徒步俱乐部去徒步了。本来之前腹部疼痛，就一直没有出门运动，医生也不能确诊，八月份做了几次检查，最后也没有什么，就让继续观察了。去的地方是暴雨顶，在京郊延庆。开始爬的时候并未觉得有多难因为是初级路线，而且天气也很好。 三小时后爬到山顶，风云变换，狂风暴雨袭来。我和老婆是不怕的，我在出门前特意塞上了冲锋衣，包里也长期放着雨衣。因为山上还有高压线，打雷期间为了安全起见，我们就在山顶的树林里避雨。我和老婆还是兴奋居多，她还在蹲着避雷的时候哼起了小曲。毕竟这是我们徒步一年以来第一次在山里忽然遇到了雨，还附赠了冰雹。领队也在观察天气和云层，几次想继续前进又担心意外所以仍旧在等待。天色十分昏暗，倒是颇有几分吓人，好在大概有十几人在一起，也算是互相支持。终于在半小时后，太阳又出现了，不过雨和雷仍然在继续，我们也趁此机会加速前进。希望可以在天黑或者天气再次恶化前下山。这次的徒步主题是观赏蘑菇，但却由于天气原因让路线强度变得升级。许多人由于经验不足，装备和体力都跟不上，导致速度变慢。雨水让下山的路也十分艰难。饶是我们穿着户外的衣服鞋子带着登山杖也是摔跤不断。领队为了保证大家的安全更改了后续的路线，下山后不再从山沟里穿越而是直接进入五公里的防火道水泥路，出去之后上马路直接就是接应的大巴车。我和夫人下山后便一路边走边聊，带着他越走越快，因为七点日落之后山里确实漆黑一片。 临出山的时候，天色擦 ...继续阅读 (6)

这本书围绕周期介绍了经济学的发展历程和关键节点和概念，对于各种流派学说也浅尝辄止。长篇的文字我以为我看不下去但是凭借着一点点的好奇和本书平实的阐述，了解了经济周期产生，发展，影响因素和核心内容。最后又结合各类资产阐述了与周期的关系。对于初学者来说是一本不错的入门书籍。 粗读完周金涛的涛动周期论对于周期产生了好奇，所以又去更基础的了解了相关的内容。虽然这本书阐述的周期是更为广泛意义上的周期，而周金涛似乎更着重研究的是康德拉季耶夫？but anyway，都很有趣。 ...继续阅读 (10)

QuestionThere are a total ofnumCoursescourses you have to take, labeled from0tonumCourses - 1. You are given an arrayprerequisiteswhereprerequisites[i] = [ai, bi]indicates that youmusttake coursebifirst if you want to take courseai.For example, the pair[0, 1], indicates that to take course0you have to first take course1.Returnthe ordering of courses you should take to finish all courses. If there are many valid answers, returnanyof them. If it is impossible to finish all courses, returnan empty array.Example 1:Input: numCourses = 2, prerequisites = [[1,0]] Output: [0,1] Explanation: There are ...继续阅读 (16)

QuestionGiven an array of meeting time intervalsintervalswhereintervals[i] = [starti, endi], returnthe minimum number of conference rooms required.Example 1:Input: intervals = [[0,30],[5,10],[15,20]] Output: 2Example 2:Input: intervals = [[7,10],[2,4]] Output: 1Constraints:1 <= intervals.length <= 1040 <= starti < endi <= 106AlgorithmA typical Chronological Ordering. We will need total order or both start and ends.While we sort start and order individually. It's fine because : we don't really care about which meeting ends and we only need to know there is a meeting ends and room are available. ...继续阅读 (11)

1248. Count Number of Nice SubarraysQuestionGiven an array of integersnumsand an integerk. A continuous subarray is calledniceif there arekodd numbers on it.Returnthe number ofnicesub-arrays.Example 1:Input: nums = [1,1,2,1,1], k = 3 Output: 2 Explanation: The only sub-arrays with 3 odd numbers are [1,1,2,1] and [1,2,1,1].Example 2:Input: nums = [2,4,6], k = 1 Output: 0 Explanation: There are no odd numbers in the array.Example 3:Input: nums = [2,2,2,1,2,2,1,2,2,2], k = 2 Output: 16Constraints:1 <= nums.length <= 500001 <= nums[i] <= 10^51 <= k <= nums.lengthAlgorithmThe nice sub array is the ...继续阅读 (14)

QuestionGiven a list ofaccountswhere each elementaccounts[i]is a list of strings, where the first elementaccounts[i][0]is a name, and the rest of the elements areemailsrepresenting emails of the account.Now, we would like to merge these accounts. Two accounts definitely belong to the same person if there is some common email to both accounts. Note that even if two accounts have the same name, they may belong to different people as people could have the same name. A person can have any number of accounts initially, but all of their accounts definitely have the same name.After merging the accoun ...继续阅读 (10)

QuestionGiven a list ofaccountswhere each elementaccounts[i]is a list of strings, where the first elementaccounts[i][0]is a name, and the rest of the elements areemailsrepresenting emails of the account.Now, we would like to merge these accounts. Two accounts definitely belong to the same person if there is some common email to both accounts. Note that even if two accounts have the same name, they may belong to different people as people could have the same name. A person can have any number of accounts initially, but all of their accounts definitely have the same name.After merging the accoun ...继续阅读 (12)

The ebook discuss things of the index , more specific , the b-tree index for developersAnatomy of an IndexAn index is a distance structure in the database that is build using theCREATE INDEXstatement ; itrequires its own disk space, hold a copy of the indexed table data and is pure redundancy-> it doesn't change the original table data and creates a new data structure that refers to the table.Searching in a database index is like searing a telephone ->the entries are ordered-> so it's fast.We cannot update the index each time the table there is a write (update/ insert) -> too slow for the quer ...继续阅读 (13)

1293. Shortest Path in a Grid with Obstacles EliminationQuestionYou are given anm x ninteger matrixgridwhere each cell is either0(empty) or1(obstacle). You can move up, down, left, or right from and to an empty cell inone step.Returnthe minimum number ofstepsto walk from the upper left corner(0, 0)to the lower right corner(m - 1, n - 1)given that you can eliminateat mostkobstacles. If it is not possible to find such walk return-1.Example 1:Input: grid = [[0,0,0],[1,1,0],[0,0,0],[0,1,1],[0,0,0]], k = 1 Output: 6 Explanation: The shortest path without eliminating any obstacle is 10. The shortes ...继续阅读 (11)

QuestionYou are given anm x ngridgridof values0,1, or2, where:each0marksan empty landthat you can pass by freely,each1marksa buildingthat you cannot pass through, andeach2marksan obstaclethat you cannot pass through.You want to build a house on an empty land that reaches all buildings in theshortest total traveldistance. You can only move up, down, left, and right.Returntheshortest travel distancefor such a house. If it is not possible to build such a house according to the above rules, return-1.Thetotal travel distanceis the sum of the distances between the houses of the friends and the meeti ...继续阅读 (10)

QuestionDesign a max stack data structure that supports the stack operations and supports finding the stack's maximum element.Implement theMaxStackclass:MaxStack()Initializes the stack object.void push(int x)Pushes elementxonto the stack.int pop()Removes the element on top of the stack and returns it.int top()Gets the element on the top of the stack without removing it.int peekMax()Retrieves the maximum element in the stack without removing it.int popMax()Retrieves the maximum element in the stack and removes it. If there is more than one maximum element, only remove thetop-mostone.You must co ...继续阅读 (17)

Self Injection!We know that there are dependency injection in Spring, who helps us to do the bean instantiation. Theself injection, you can tell from its name, is a bean inject itself in its class, to overcome the limitation described below of AOP.WhyStart from @Transactional```java @Service public class MyService { public void doSomething() { // ... doSomethingElse(); }@Transactional public void doSomethingElse() { // ... }} ```WhendoSomething()is called, the transactional annotation doesn't take effect. -> when you call another method in a method(self invoking met ...继续阅读 (10)

Chapter10 Batch ProcessingService (only system ): wait request from client to processBatch : takes large amount of data and process and output some dataStream (near real-time ): something in the middleBatch Processing with Unix ToolsSimple Log AnalisysThe log is a the request log for a server , this can help find the top 5 most popular websiteChain of commands versus custom programYou can also write small scripts to do this.ruby counts = Hash.new(0) File.open('/var/log/nginx/access.log') do |file| file.each do |line| url = line.split[6] counts[url] += 1 end end top5 = counts.map{|url, coun ...继续阅读 (12)

Chapter9 Consistency and ConsensusWe need to build systems that is fault tolerant -> can be with fault but still aliveThe best way of building fault tolerant system is to find some general purpose abstraction with useful guarantees , implement once and let the system rely on those grantees.Consensus : all nodes agree on something.e.g. in single leader replication , all nodes need the consensus that : there should be only one leader in the systemConsistency GuaranteesIn replication, data reach to different node at different time -> you may read different data at a time point from different node ...继续阅读 (20)

Chapter 8 The Troubles with Distributed SystemsFaults and Partial FailuresIn single machine, Good software /systems are either fully functional for entirely broker, but not something in between. -> we don't want the faulty result which cause useless work and confusingThe nondeterminism an possibility of partial failure is what makes distributed system hard to work with -> some times it works well but sometime not, you sometime event don't know the failure due to the nondeterministic network failureCloud Computing and SupercomputingLarge scale systemsOne end is the field of high-performance co ...继续阅读 (19)

Chapter7 TransactionA transaction is a way for an application to group several reads and writes together into a logical unit.Either the entire transaction succeed(commit); or it fails (abort, rollback)The Slippery Concept of a TransactionThe meaning of ACIDAtomicity, Consistency, isolation and durabilityAtomicitySometing cannot be breakdown to smaller partsIn ACID context , atomicity describe what happens is a client wants to make several writes but a fault occurs after some of the writes have been processed.Without it, some changes may take effect but we cannot know and when there is a retry, ...继续阅读 (15)

Field injection? Constructor injection? Or setter injection?What are theyDifferent flavors ofdependency injection```java class MyComponent {@Inject MyCollaborator collaborator;public void myBusinessMethod() { collaborator.doSomething(); } } ```DI is the means to connect the two.Why field injection is evilField injection allows dependencies to be null by defaultHere is another way of injection:```java class MyComponent {private final MyCollaborator collaborator;@Inject public MyComponent(MyCollaborator collaborator) { Assert.notNull(collaborator, "MyCollaborator must not be null!"); this.co ...继续阅读 (14)

Chapter6 PartitioningScalability is the main reason for partitioning : different partitions can be places on different node in a shared-nothing cluster . Thus, large dataset can be distributed across many disks and the query load can be distributed across many processors.This chapter will discuss ways of partitioning , and how the indexing interacting with data partitioning ; then talk about the rebalancing if you want to add or remove nodes in the cluster; finally we discuss how databased route requests to the right partitions and execute queries.Partitioning and ReplicationPartition is usua ...继续阅读 (14)

Replication of database helps in many aspectsScalability -> spread increasing data loadFault tolerance/high availability -> one down, another take over;Latency/performance-> multiple datacenter across world might help avoid network travel across the earth;Scale to higher load:Shared memory (powerful machine): cost grow fast; limited fault tolerance -> vertical scalingShared disk: several independent CPU and a shared disk(they are connected via network): may suffer from lockShared nothing : horizontal scaling , has many nodes(machines) with their own CPU and disks;Data being distributed across ...继续阅读 (14)

Chapter4 Encoding and EvolutionIn this chapter, we will explore how several formats handle schema change and how they support system where old and new data coexist.How these formats are used for data storage and communicationFormates for encoding DataPrograms usually work with data in two different representations: one is in-memory data structure arrays, maps...., those structure are optimized for CPU to handle with pointer; the other one is byte(like JOSN document) that used for data transformation in network or file system.They need to be somehow transferred from each other :In-memory-> byt ...继续阅读 (12)

Chapter3 Storage and RetrievalData structures that power your databaseLog based database needs index to find particular data efficiently.The idea behind is basically keep some additional data aside , which helps you locate data you want.Adding index incurs overhead of writing, but boots readingApplication engineer add index wisely .Hash IndexesK-V index can be implemented by hash map.e.g. you have a in memory hash map that stores the key and offset of the data for that key. You don't have to go through all the file(log-structured file on disk), you only refer the map and go to the right positi ...继续阅读 (14)

Chapter2 Data models and Query LanguagesData models effects how your software is written, but also how we think about the problem we are solvingApplication engineers model real-world objects to programming objects and data structures. (API layer)Lower level, when you want to store these objects, they are expressed in general-purpose data model, JSON,XML, tables in relational db or graph model.Database engineer decides ways of representing those data in terms of bytes in memory, disk or on a networkLower level hardware engineers figure out how to represent bytes interns of electric currents, pu ...继续阅读 (12)

Chapter1 - Reliable, Scalable and Maintainable applicationThis chapter introduces the terminology and approach we used in the book; and gives examples in a high level of these terminology(what and how to achieve)Data-intensive application is build from standard building blocks :Databases -> store data that can be used(read) againCaches -> speed up reads (remember the result of expensive operation )Search index -> allow users to filter or search by keywordStream processing -> handle data asynchronouslyBatch processing -> periodically crunch a large amount of accumulated dataThese blocks are hig ...继续阅读 (14)

Protect Clients from serversSynchronous and asynchronous clientsAdmission control systemBlocking I/O and non-blocking I/O clientsAdmission control system : system with load shedding and rate limiting -> back exception -> HTTP 429 and 503Client should avoid immediate retry -> retrying with exponential backoff and jitters is betterDelays give servers to scale outJitters distributes request more evenlyWhat if servers are not elastic and the problem is not transient ?Blocking / non-blocking client (sync and async )Blocking:if the thread want to send requests to server, it starts new connection or ...继续阅读 (12)

描述在数据更新操作时，希望利用分布式锁-乐观锁机制预防并发修改，增加了对于entity的version的update以引起乐观锁版本的检查。实际的现象是修改可以正常保存，set 的version被忽略，直接在根据数据库中的数据做了递增。原因JPA/Hibernate中的实体有四种状态org.hibernate.event.internal.EntityState{PERSISTENT,//托管状态TRANSIENT,//瞬时状态DETACHED,//游离状态DELETED;//删除状态}当除以一个事务中时， 查询出的实体将一直处于PERSISTENT状态，对这个状态下的实体修改不会检测乐观锁，手动setVersion也会被忽略。所以要让实体不在处于PERSISTENT状态。当查询与保存不在同一事务时，实体由于查询事务的结束，将变成DETACHED状态，这对JPA来说，该实体是不可信的，所以对实体保存时，JPA将通过主键再次查询数据库，进行数据比对，这也包括乐观锁的检测。 同样new一个实体时，也不会处于PERSISTENT状态， 这也可以迫使JPA进行乐观锁检测。解决方法将本次操作的查询与保存放到俩个事务中；实物保持不变，不用查询出来的实体进行set操作，new一个实体，将查出的实体与请求提交的实体合并，然后保存。（利用merge）JPA/Hibernate 代码实现org.hib ...继续阅读 (15)

Protect Servers from clientsSystem OverloadWhy it's important to protect a system from overloadAuto scalingScaling policies(metric-based, schedule-based, predictive)Overload happens when existing computational resources are not enough to process all the loads;-> add more servers manually?Do it often is operationally expensiveInfrequent changes increase the likelihood of overloadLeave it excess the needs is is waste of costWe should find ways to do(add and remove) it automatically -> auto scalingImproves availability -> system add or remove machines on the unpredictable traffic spikesReduce cos ...继续阅读 (15)

PartitionHow to scale message consumptionSingle consumer vs multiple consumersProblems with multiple consumers(order of message processing, double processing)Scale horizontally: more machine or vertically better machineSingle consumer: message order is preserved but unreliable(restart, network issue...) ;Multiplen consumers : high availability ; but messages can be processed out of order; and higher change of processing the same messages multiple time.Out of order issue demonstration: push model:I have a queue and two consumers 1 and 2 messages are dispatched in a round robin manner ;I have t ...继续阅读 (13)

Deliver data reliablyWhat else to know to build reliable, scalable and fast systems?List of common problems of distributed systemList of system design concepts that can help solve these problemsThree-tier architectureProblems in distributed systemsTransient network issuesLong lasting network useConsumer application failures-> broker may keep the messages for a while until its disk is full;Timeout, reties, idempotency, backoff with jitter, failover fallback, message delivery guarantees, consumer offsetsIf messages comes faster than then speed of consumer can process them due to -> traffic burst ...继续阅读 (16)

How to deliver data quicklyBatchingPros and cons of batchingHow to handle batch requestsBatching : group messages into a singe requestPro:Increase throughput: less requests , less overhead , less connectionsDecrease cost(especially for cloud)Cons:Complex in sender and receiverSender: takes time buffer messages and send out based on time or size; can be hard to implement and configureReceiver: process message one by one but what if one message fails? Roll back all of them and let sender send again to reprocess? If not how do we let sender to send the fail messages ?How does server handle batch ...继续阅读 (15)