Just write this snippet for my practice of 0-1 Knapsack Problem:values = [1, 2, 3, 4, 5] weights = [3, 2, 1, 9, 6] max_weight = 12 def knappack(): n = len(values) dp = [[0] * (max_weight+1) for _ in range(n)] print(dp) for i in range(n): for j in range(1, max_weight+1): if weights[i] > j: dp[i][j] = dp[i-1][j] else: dp[i][j] = max(dp[i-1][j], dp[i-1][j-weights[i]] + values[i]) print(dp[-1][-1]) knappack() ...继续阅读 (105)

After reading theclassic paperabout Skip-list, I tried to implement it by myself. But then I found a line of pseudo-code in the “Delete” function that I couldn’t understand:Seems all the elements in “update” will point to “x”, so why do we need to check here? Maybe I can ignore this checking. Now here comes a part of my code:def erase(self, num: int) -> bool: update = Node(-1) curr = self._head for level in range(MAX_LEVEL-1, -1, -1): while curr._forward[level] and curr._forward[level]._key < num: ...继续阅读 (111)

This paperintroduced a method to extract only segments with bird sound from an audio file. Since the paper didn’t give any code, I started to write it by myself.Here is the Python implementation:import cv2 import time import torch import librosa import soundfile as sf import numpy as np from torchlibrosa.stft import LogmelFilterBank, Spectrogram class CFG: n_fft = 2048 hop_length = 512 sample_rate = 32000 n_mels = 64 fmin = 150 fmax = 150000 class SignalExtractor: def __init__(self): self.spectrogram_extractor = Spectrogram( n_fft=CFG.n_ ...继续阅读 (116)

Trying to test theIntel extension for PyTorchin my project, but it reported errors:Traceback (most recent call last): File "reviewjpgs_optimaztion_testing.py", line 27, inimport intel_extension_for_pytorch as ipex ...继续阅读 (100)

I created a simple Argo job to pull messages from a Google Cloud Pub/Sub topic. Permission has been given to the service account of GKE’s workload identity. But the Argo job failed with errors:argo submit example.json -n argoprojhello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable hello-world-pqbm5: return callable_(*args, **kwargs) hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/grpc/_channel.py", line 1030, in __call__ hello-world-pqbm5: return _end_unary_response_blocking(state ...继续阅读 (105)

I want my Python script to receive one message from a PubSub topic and then go on to other work. The code is learned from an example of the GCP document:with subscriber: # The subscriber pulls a specific number of messages. The actual # number of messages pulled may be smaller than max_messages. response = subscriber.pull( request={"subscription": subscription_path, "max_messages": NUM_MESSAGES}, retry=retry.Retry(deadline=300), ) if len(response.received_messages) == 0: returnThe problem is that it will receive empty messages, meaning that “l ...继续阅读 (119)

Long story short. I am trying to build aSiamese networkfor audio classification. For 50% possibility, the “dataset.py” will try to find a pair of audios in the same category but with different files (also, different category for another 50% possibility). But when the evaluating start, it will hang after fetching a few batches. The trace could be see:Traceback (most recent call last): ...继续阅读 (111)

In the research of PCIE 3.0 versus PCIE 4.0, I became serious about the actual application scenario. What’s the real bandwidth between CPU and GPU when we are training a deep learning model?Finally, I got this tool:pcmAfter building it, I run “sudo ./bin/pcm” and got this:Grateful that I can even see the IPC(Instructions Per Cycle), L2/L3 hit ratio from this tool. But my most interested metric is the PCIE bandwidth. Where is the PCIE bandwidth?I tried “sudo bin/pcm-pcie” but it said my desktop CPU (i5-12400) is not supported:The processor is not susceptible to Rog ...继续阅读 (104)

My first idea is depth-first-search: iterate all people, try to give them different hats. The solution got TLE (Time Limit Exceeded). Then as a hint from discussion forum, I started to iterate hat (instead of people), try to give them different people. The solution also got TLE (even I used lru_cache for function):from collections import defaultdict class Solution: def numberWays(self, hats: List[List[int]]) -> int: hp = defaultdict(set) for index, hat in enumerate(hats): for _id in hat: hp[_id].add(index) hp = ...继续阅读 (109)

After installing an RTX 2080 TI on an old-2016-desktop at the beginning of 2019, we used it to trainYOLOv6for a while. But recently the training job will occasionally hang and the GPU stops working. The only message I can see is fromdmesg[ 8104.078794] NVRM: GPU at PCI:0000:01:00: GPU-b4f425ef-2d0f-f29e-5624-ff96b37c2c46 [ 8104.078796] NVRM: Xid (PCI:0000:01:00): 79, pid='', name=, GPU has fallen off the bus. [ 8104.078797] NVRM: GPU 0000:01:00.0: GPU has fallen off the bus. [ 8104.078803] NVRM: A GPU crash dump has been created. If possible, please run NVRM: nvidia-bug-report.s ...继续阅读 (103)

To solve Leetcode #1675, I wrote the below code with the help of hints:import sys import heapq class Solution: def minimumDeviation(self, nums: List[int]) -> int: n = len(nums) heapq.heapify(nums) _max = max(nums) ans = max(nums) - min(nums) while True: item = heapq.heappop(nums) if item % 2 == 1: heapq.heappush(nums, item * 2) _max = max(_max, item * 2) ans = min(ans, _max - heapq.nsmallest(1, nums)[0]) else: heapq.heappush(nums, item) ...继续阅读 (112)

The first intuitive idea that jumps out of my mind after taking a look at LeetCode #127 is using the graph algorithm. But for building the graph first, I need to traverse thewordListby O(n2) times.Here comes the time complexity analysis: the length of thewordListis about 5000, O(n2) means 5000*5000=25*106. For a python script in LeetCode, it will cost about 1 second for every 106operations. Thus 25*106will cost about 25 seconds, which is too long for a LeetCode question.Therefore the best method to build a graph is not to traverse the wordList multiple times, but to just iterate all lower-case ...继续阅读 (118)

The popular solution for LeetCode#494is dynamic programming. But my first idea is Divide and Conquer. Although it’s not very fast, it’s another idea:from collections import Counter class Solution: def get_results(self, nums: List[int]) -> Counter: layer = [nums[0], -nums[0]] for num in nums[1:]: new_layer = [] for item in layer: for val in [-num, num]: new_layer.append(item + val) layer = new_layer return Counter(layer) def findTargetSumWays(self, nums: List[int], target: i ...继续阅读 (101)

The first idea that jumped out of my mind was using Sets to track two nodes and pick up the first intersection node between these two Sets. Hence came out the first solution:from collections import defaultdict class Solution: def bfs(self, node1: int, node2: int, conns, length) -> int: set1 = {node1} set2 = {node2} step = 0 while step <= length: inter = set1 & set2 if len(inter) > 0: return min(list(inter)) new_set1 = set() new_set2 = set() for node in set1: new_set1 ...继续阅读 (99)

My first solution is using dynamic programming. Then I want to also try breath-first-search. The first version of my BFS:class Solution: def coinChange(self, coins: List[int], amount: int) -> int: if amount == 0: return 0 # bfs depth, ans = 1, -1 queue = [amount] while len(queue) > 0: new_queue = [] for node in queue: for coin in coins: if coin > node: continue if coin == node: return depth el ...继续阅读 (102)

We were usingclient.query()(from Python API of BigQuery) to insert selected data into a table with a specific partition. But the script reported errors like:google.api_core.exceptions.BadRequest: 400 Some rows belong to different partitions rather than destination partitionThisnotesaid it might be the cause of the incorrect date format for the partition. I checked the code but only found the partition format is correct.The real reason is the input: the “selected data”. The data that will be inserted is from this SQL:SELECT col1, col2, "2023-01-06" as partition_date FROM my_table;Th ...继续阅读 (97)

It does not mean I just read three books in the whole year of 2022 when I just showed three books above. Although reading some short history stories and books about financial knowledge, I still think those are not vital to my life experience.As a software developer, why did I read a book about the semiconductor material: Silicon Carbide? Frankly speaking, just because of curiosity. In recent years, a lot of news and articles said SiC (Silicon Carbide) will be the future material of EVs (Electric Vehicles), and all the companies that produced SiC will become extremely popular and rich. But afte ...继续阅读 (109)

pendulumis a prevalent python library in my company. For example, if I want to get the time of the previous Monday, it could be written:pendulum.today("US/Pacific").previous(pendulum.MONDAY) # return type "pendulum.datetime.DateTime`But this will return a class ofpendulum.datetime.DateTime. What if I wishDateinstead ofpendulum.datetime.DataTime? Actually, thependulum.datetime.DateTimeinherited fromdatetime.datetimeso we can use.date():pendulum.today("US/Pacific").previous(pendulum.MONDAY).date() # return type "pendulum.datetime.Date` ...继续阅读 (99)

My configuration DAG ofArgoworkflow was like this:{ name: "my-workflow", dag: { tasks: [ { name: "step1", template: "template1", }, { name: "manual-check", template: "template-manual-check", depends: "step1.Failed", }, { name: "step2", template: "template2", depends: "manual-check", }, ], }, }If “step1” failed, the “manual-check” will suspend the whole pipeline and let users (or customers) decide whether this pipeline could continue. But I met a funny situat ...继续阅读 (99)

Here is the code for me to query a table of BigQuery:from google.cloud import bigquery from google.cloud.bigquery_storage import BigQueryReadClient client = bigquery.Client() storage_client = BigQueryReadClient() df = client.query("select * from my_table1").to_dataframe(bqstorage_client=storage_client)Then it reported the error:“Access Denied: Project PRJ_B: User does not have bigquery.jobs.create permission in project PRJ_B.”But actually, I want to launch a job in project PRJ_A. So I add a shell command “gcloud config set project PRJ_A” before running this python scri ...继续阅读 (93)

When using “terragrunt init”, an error jumped out:DEBU[0002] Remote state GCS bucket prokyon-systems-state-bucket does not exist. Attempting to create it prefix=[/data/proj/prokyon-systems/auto-accountant/infra/non-prod/europe-central/dev/storage-bucket] ERRO[0002] Missing required GCS remote state configuration project ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1This error looks a little confusing because my attention was completely drawn by the “remote state configuration”. Where could I find the “remote state ...继续阅读 (97)

I already knew that Vertex AI is a new product of GCP so it tends to be unstable. But I didn’t imagine it lacks a vital function: log errors or exceptions thrown by python code.After I added code fromthis article, some of my steps, fortunately, reported error logs. But for some other steps, they jump out with just a simple message:You may think I could find more detail in the URL it gives at the end. But that’s not true. After I copy and jump to that URL. It showed nothing:Don’t be confused. The blank above is from the GUI of GCP because there is no record.If a pipeline frame ...继续阅读 (97)

Here is my code and it caused OOM (“Out Of Memory”) when running:for img_batch in data_load: input = torch.from_numpy(np.asarray(img_batch)).cuda() result = self._net(input.permute(0, 3, 1, 2).float()) values, indices = torch.topk(result, 10) for index in range(len(values)): top10 = values[0] statistics["accumulate"] += top10[0]It firstly caused CUDA to report OOM so I just stupidly removed the “cuda()” to let inference run only on the CPU.But quickly, the CPU program also reports OOM. And this time I realised that the variable “top10” ...继续阅读 (96)

I know that we can directly use “results.show()” to get the image with objects being figured out. But what if I want to just show some objects that have bigger confidence than a threshold? Then we need to fetch the results one by one manually:import torch import cv2 model = torch.hub.load('.', 'custom', path='best.pt', source='local') model.eval() colors = { 14: (0,255,0), 80: (0,0,255) } names = { 14: "bird", 80: "squirrel" } for index in ["1.jpeg", "2.jpeg", "4.jpeg", "7.jpeg", "3.webp", "5.webp", "6.webp", "8.png"]: img_name = f"squirrel_bird{index}" ...继续阅读 (95)

To detect birds and squirrels, we created adatasetto train theYOLOv5model. After a week’s training with:python3 -u train.py --data coco.yaml --cfg yolov5s.yaml --weights '' --batch-size 28 --workers 1The model could recognize birds and squirrels properly except only for this image:Why does the model recognize the right-side significant squirrel as a bird? Even though I tried a bigger model, the result was the same…Only after researching the parameters of the functionmodel()of YOLOv5, I found out we can use a different image size: 960 for detecting.import inspect import torch import ...继续阅读 (141)

Two days ago we met a weird error when running aselectthrough BigQuery Python API:Error : google.api_core.exceptions.BadRequest: 400 Bad int64 value: BA1DI checked theselectSQL but it doesn’t contain any type like “int64”.After “binary search” in the SQL code, I finally found out that the SQL is actually querying a “view” and the code of this view is like:SELECT cast(col1, int64) AS COL1, cast(col2, int64) AS COL2, FROM table1The correct solution is to change “cast” to “safe_cast”.Here is the lesson for me: some errors may ...继续阅读 (91)

I have been usingRegNetYinDongNiaofor almost two years. Previously I was just using small models such as RegNetY-8G. But after having a computer with RTX-3080-TI, I started to use the biggest one in the original paper — RegNetY-32G.RegNeyY-32G model costs a lot of time for training so I would usemixed-precisionin the process. However, after using “float16”, the training program always crashes with the error of overflow:... Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 2.0237e-320 Gradient overflow. Skipping step, loss scaler 0 ...继续阅读 (91)

apexis a mixed training library from Nvidia. I have been using it since I got an RTX3080TI GPU. A few days ago, I started to useRegNetY-32GF (I just used RegNetY models smaller than16GF previously). After a accidental break, I tried to resume the training but it reported:Traceback (most recent call last): File "train.py", line 353, intrain(args, train_loader, eval_loader) File "train.py", line 220, in train scaled_loss.backward() File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 401, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph ...继续阅读 (96)

Dong:http://oldblog.donghao.org/2007/09/euoo.html你说时间过得多快Hero: ...那时怎么大家这么无聊无聊得来有基情http://oldblog.donghao.org/2008/02/io.html唐唐是谁？Dong: 唐勇，2008年我挪到他下面，就没有跟中中在杭州一起加班了“要求英俊潇洒、玉树临风、还要懂c++、懂后台系统”，现在公司里满足这个要求的就多多了，还是钱多雇得起啊Hero: 哦哦Dong:http://oldblog.donghao.org/2007/09/u.html这个才是境界Hero: ...真无聊关于我的有这么多？还有呢Dong: 无聊的另一个意思就是：禅
。你计划书上要注意用词Hero: 有道理，英文是zenDong:http://oldblog.donghao.org/2007/09/euoaee.html主要是你这位大师在制造zenHero: ...Dong: 不过最近浏览论文才发现，CNN居然是多伦多大学的一位教授最早提出的，此人现在还在从事深度学习方面的研究，论文也还在发。说真的，你顺路可能去拜访他吗？Hero: 是啊，很棒的啊，人被 google 收购了拜访他干嘛啊Dong: 问得好，看来我得研究明白了，然后发两篇有价值的论文，然后让公司赞助我去多伦多开个学术会，然后我就能当面请教和沟通了，比单 ...继续阅读 (95)

上周五去北航参加了一个文件系统和存储的交流会，是华为牵头办的，小会议，参会不到30个人。本人水平有限，只能挑一些听懂的留个笔记。首先是清华大学的陆游游同学讲一个全新设计的基于FLASH存储的文件系统。大概思路是将存储管理和SSD上的FTL合并在一起做成一层，还利用NAND片上每个PAGE的额外存储空间存放一些元数据以在机器断电重启后重新沿链接找回数据（这样就可以不用journal了，似乎也可以认为是把journal放在了PAGE的额外存储空间里），然后在上面架一个新的简化的文件系统做基本的namespace管理。这个思路跟DFS有点像，不过DFS只是上面那一层薄薄的namespace管理，下面就交给Fusionio的firmware了。华为的谢美伦介绍了手机终端上几种常用文件系统的评测。Android上当然首推ext4，但是后起之秀f2fs更简单且随机写性能不输ext4（这点大家有争论，有人认为这个评测应该写满存储卡再全删掉，再重新测试，也就是要把FTL打乱了再测，会更公正），不过来自阿里云的刘铮表示：google倾向于在服务器和手机上使用统一的文件系统即ext4以方便维护。看来f2fs虽然更贴合NAND，但是大厂支持上还不够丰富。来自百度的杨勇强介绍了百度的分布式存储和文件系统。这个，互联网的玩法太相似了，主从结构，erasure code，就不累述了。讲到中途的时候，来自华为存 ...继续阅读 (98)