Just one picture:I don’t know why the editing place of App name is under “Grow users”. But unfortunately, it’s there.After you change the “App name” and click “Save” (You also need to upload a bunch of images before click it. Damn it) ...继续阅读 (8)

I found a very interesting picture:The size of this image is about 8MB although it’s blurring. Then I use below python code to try to resize it using different interpolation strategy:import cv2 img = cv2.imread("/Users/robin/Downloads/ou1.png") for inter, name in [(cv2.INTER_AREA, "area"), (cv2.INTER_CUBIC, "cubic"), (cv2.INTER_LINEAR, "linear")]: after = cv2.resize(img, (310, 310), inter) cv2.imwrite(f"{name}.bmp", after)“area”“cubic”“linear”All these resized pictures looked terrible.Is this the general problem of Image Resizing? Let me try ...继续阅读 (8)

After I installed two versions of JDK (17 and 21) and uninstalled them, I saw this error when trying to launch my Android Studio. This error is hard to fix. Reinstalling Android Studio won’t fix it. And even after I searched google and asked chatGPT and try all the suggestion they give, the problem continued for two days.The eventual solution is from a StackOverflow article (sorry I forget the url):# firstly, manually uninstall Android Studio # then rm -rf /Library/Java/JavaVirtualMachines/* # then rm ~/Library/Application\ Support/Google/AndroidStudio* # then rm -rf ~/Library/Caches/ # ...继续阅读 (10)

After training both image classification and sound classification deep learning models. I found out that the image training is much slower than the sound training, although the sound dataset is much bigger than image dataset.The first idea that jumped out of my mind is that the image has 3 channels (RGB) but the sound spectrogram just have 1 channel. Therefore if I compress RGB into 1 channel (such as using gray image), the training speed of image classification will become 3 times faster.A few days ago, I started to train the image classification with gray image. But the speed of training is ...继续阅读 (15)

...继续阅读 (12)

All the code ishere.Thebaselineof training balanced data ofAudioSetis 0.27 mAP. Using TimeMasking and FrequentMasking could slightly push it to 0.28 mAP.I tried mixup of raw sounds likeASTbut it didn’t improve the mAP totally (the reason is still a myth for me). But, the mixup of fbank filters could push metric to 0.293 mAP.Until then, the fbank filter will be resized to (384, 384) for modeldeit_distilled. After Irecoveredthe size of fbank filter to (128, 998), it reached 0.323 mAP.The most recent (hope it’s not the last)changeis copied wholly fromAST: use the pretrained parameters ...继续阅读 (39)

I was trying to implementALBEFbymyselffor practice. After finishing all the parts (Vision part, BERT part, including Masked Language Model), I trained the model on COCO-Captions/SBU-Captions/CC3M/CC12Mdataset (actually more data than the original paper). But the result is quite weird. An old steam train was recognised as a building, and a few fish were recognised as statues.To solve these weird mistakes, I reviewed the code many times and finally noticed a sentence in the paper:Although it’s just a normal sentence in the paper, the augmentation could improve the ALBEF model significantly ...继续阅读 (37)

To create a pipeline schedule of Vertex AI, we can usebelow snippet:from google.cloud import aiplatform pipeline_job = aiplatform.PipelineJob( template_path="COMPILED_PIPELINE_PATH", pipeline_root="PIPELINE_ROOT_PATH", display_name="DISPLAY_NAME", ) pipeline_job_schedule = pipeline_job.create_schedule( display_name="SCHEDULE_NAME", cron="TZ=CRON", max_concurrent_run_count=MAX_CONCURRENT_RUN_COUNT, max_run_count=MAX_RUN_COUNT, service_account="XYZ", )This Python code runs with service account “XYZ” and we also want the schedule to run as service account “XYZ& ...继续阅读 (58)

When joined the Junior Middle school in a small town in west-south China in 1993, I met my first English book. Yes, it looks exactly like this:Then, the terrible 6 years of Chinese-style-English-learning started. For normal kids in poor regions of China, the only way to learn a new language is to REMEMBER IT. For the vocabulary, I remembered all of them by writing them in the draft paper again and again. For the phonogram, I remembered all of them by writing them in the draft paper again and again. For the grammar, I remembered all of them by —- Wait a minute. Why did I need to remember ...继续阅读 (54)

I just wrote my implementation ofALBEFin my own way. But when evaluated with some masked sentences, it failed.I am using this image:When I asked “This is a chocolate <|mask|>”, it generated “This is a chocolate urn”. Quite strangeThen I asked “This is a <|mask|> cake, it generated “This is a iph cake”. Totally wrong.After checking my implementation of the dataset, and training on a small part ofCC3M, a week passed and I finally got the reason today: thetiktokenis a BPE tokenizer that will use sub-words as tokens and these sub-words severely hurt the mo ...继续阅读 (53)

Three weeks passed since theprevious article. Here are the answers to the previous three questions:Q1: The original paper of CLIP uses L2 normalization on multimodal embeddings. But in my test, the model will not converge with it.Answer 1: The reason the model didn’t converge is that the learning rate is too large. After reducing the learning rate a bit and adding the L2 normalization, the model could get above 80% validation accuracy. The L2 normalization essentially projects embeddings to a high-dimensional sphere with a one-unit radius, which by intuitive could regularize the model.Q2 ...继续阅读 (67)

After downloading the whole CC12M dataset from Huggingface, I wrotea toolto extract all of the image-text-pair files into one directory. But after extracting 17 million (17681010 exactly) files, the tool reported the error:Exception: [Errno 28] No space left on device: '/home/robin/Downloads/cc12m/011647171.txt'I checked the space and inodes in my ext4 filesystem, and seems they all have free capacity:# "df -lh" Filesystem Size Used Avail Use% Mounted on tmpfs 3.2G 2.0M 3.2G 1% /run /dev/nvme0n1p1 916G 410G 461G 48% / tmpfs 16G 412K 16G 1% /dev/shm tmpfs ...继续阅读 (59)

The GPT part of myMultimodal trialsmainly comes fromnanoGPT. In the nanoGPT, the Positional Encoding is just a learnable tensor (“wpe” means “weights of positional embedding”):self.transformer = nn.ModuleDict(dict( wte = nn.Embedding(config.vocab_size, config.n_embd), wpe = nn.Embedding(config.block_size, config.n_embd), drop = nn.Dropout(config.dropout), h = nn.ModuleList([Block(config) for _ in range(config.n_layer)]), ln_f = LayerNorm(config.n_embd, bias=config.bias), ))It’s different from the ...继续阅读 (59)

CLIPis already a three years old paper but its simple design and significant performance still attracted me. After one week of programming and debugging, I finishedv0.1-versionof my tiny CLIP. It uses ConvNextV2 Nano and some parts of nanoGPT so both encoders will keep parameters of about 35millons.The training dataset isCC3Mdownloaded by using the tool fromimg2dataset. The actual number of images is 2.3 million (might be my awful network environment). For the testing dataset, I use the 50000 val images ofImageNet1K.I split the CC3M into 90% training and 10% validating. Just after one night o ...继续阅读 (55)

I am trying to build a smallrepoabout multi-modal models (CLIP, ALBEF, BLIP etc). The GPT code is mainly fromnanoGPT. Then I became inquisitive about the performance of “Flash Attention” and “torch.compile()”.The metrics with my original code (w/o Flash Attention, w/o torch.compile()):[100] loss: 4.0315 time 23.7708 [200] loss: 4.0020 time 23.9010 [300] loss: 3.8115 time 23.9407 [400] loss: 3.7021 time 23.9785 [500] loss: 3.6626 time 24.0076 [600] loss: 3.7109 time 24.0060The metrics after adding Flash Attention:[100] loss: 4.1204 time 23.0655 [200] loss: 3.8950 time 23 ...继续阅读 (89)

deepspeed \ –master_addr=rogpt1 \ –elastic_training \ –min_elastic_nodes=1 \ –max_elastic_nodes=2 \ –hostfile=hostfile \ train.py \ –deepspeed_config ds_config.jsondeepspeed \ --master_addr=rogpt1 \ --elastic_training \ --min_elastic_nodes=1 \ --max_elastic_nodes=2 \ --hostfile=hostfile \ train.py \ --deepspeed_config ds_config.jsondeepspeed \ –master_addr=rogpt1 \ –elastic_training \ –min_elastic_nodes=1 \ –max_elastic_nodes=2 \ –hostfile=hostfile \ train.py \ –deepspeed_config ds ...继续阅读 (59)

Let’s get to the point directly:import os import time import torch import torch.nn as nn import torch.distributed as dist from model import resnet152 from dataset import get_data_loaders from torch.nn.parallel import DistributedDataParallel as DDP learning_rate = 0.001 num_epochs = 40 momentum = 0.9 weight_decay = 1e-5 def setup(): # initialize the process group dist.init_process_group("nccl") def cleanup(): dist.destroy_process_group() def train(rank, world_size): setup() model = resnet152().to(rank) model = DDP(model) if rank == 0 and os.path.exis ...继续阅读 (65)

We have a project using Pub/Sub of Google Cloud. About one month ago, the pipeline failed because the subscription inexplicable disappeared. I suspected someone may mistakenly deleted it. However, after searching for a mount of log of GCP, nothing has been discovered. Without any clue, I just re-created the subscription.Then comes the new year. The subscription disappeared again. This time, it looks like a system setting instead of a human mistake. Fortunately, we found this doc from GCP:https://cloud.google.com/knowledge/kb/pub-sub-subscriptions-disappeared-without-any-deletion-logs-000004170 ...继续阅读 (62)

Here is the original picture:Months ago, I tested thesegmentation of YOLOv8. The result is not very promising:The tail of one monkey couldn’t be segmented correctly.Today I tested the same picture with Meta’sSegment Anything Model(SAM). After using the simpleColab notebookand “vit_l” model type, the result shows better:At least the yellow monkey’s tail has been all correctly segmented.Note: For running the notebook with T4 GPU, you may need to setpoints_per_batchlike:SamAutomaticMaskGenerator(sam, points_per_batch=16)How about running with another picture? The goa ...继续阅读 (89)

If we want to use YOLOv8 for object detection, here isa good example.What if I want to use YOLOv8 to segment a picture, crop out the object, and paste (only paste the object, not the pixels near it) it to a new picture? I wrote an example:import cv2 import numpy as np from ultralytics import YOLO IMG_SIZE = 2048 def main(path): filename, file_extension = os.path.splitext(path) # Load segment model of yolov8 model = YOLO("yolov8x-seg.pt") img = cv2.imread(path) results = model(img, imgsz=(img.shape[1], img.shape[0])) count = 0 for res in results: for ma ...继续阅读 (86)

Last week I noticed that the pod in Argo would give a UTC timezone even though the Argo configuration has set a AEDT timezone.apiVersion: argoproj.io/v1alpha1 kind: CronWorkflow metadata: name: my-cron-workflow namespace: MYNAMESPACE spec: schedule: "0 10 * * *" timezone: "Australia/Sydney"Actually, thistimezonein configuration is only used to give the time this scheduled job would run at, not the timezone in every pod.To set up a unified timezone for all the pods, we need to usevolume.apiVersion: v1 kind: Pod metadata: name: busybox-sleep spec: containers: - name: busybox im ...继续阅读 (87)

I was trying to launch a VM instance with GPU on Google Cloud. But after trying T4, L4, and V100, they all reported “exceeding resource limit”, which means a lot of people in my region are using these types of GPUs.Without choice, I launched a VM instance with an old Nvidia Tesla P100 (I first used it about 5 years ago). Then, I need to install itsdriver. But the installation process reported errors:*** Failed CC version check. *** SYMLINK /tmp/selfgz26389/NVIDIA-Linux-x86_64-515.105.01/kernel/nvidia/nv-kernel.o SYMLINK /tmp/selfgz26389/NVIDIA-Linux-x86_64-515.105.01/ker ...继续阅读 (92)

Just using the Python API of Text-to-Speech AI to transform a PDF file to mp3 audio, as theexample:from google.cloud import texttospeech from PyPDF2 import PdfReader client = texttospeech.TextToSpeechClient() reader = PdfReader("xxx.pdf") voice = texttospeech.VoiceSelectionParams( language_code="cmn-CN", name="cmn-CN-Wavenet-B", ssml_gender=texttospeech.SsmlVoiceGender.MALE ) audio_config = texttospeech.AudioConfig( audio_encoding=texttospeech.AudioEncoding.MP3, speaking_rate=0.8, ) text = "" index = 1 # try first 10 pages for page in reader.pages[:10]: text += page.extrac ...继续阅读 (100)

Although having used the YOLOv5 model several times, I haven’t used its corresponding ONNX model before. This time, I met a use case to run its ONNX model using Javascript.To learn and debug the code, I installed node.js and started my Javascript trip. Thissnippethelps me a lot to understand the Non-Max Suppression algorithm again (I used NMS algo many years). After studying it and also theimplementationin the YOLOv5, I finally knocked together the Javascript code to use the ONNX model of YOLOv5 to detect objects:https://github.com/RobinDong/onnx_js/blob/main/detect.jsWhy the “YOLO ...继续阅读 (93)

audiomentionsis a very convenient library for my bird sound classification. As the code below:from audiomentations import Compose, AddGaussianNoise, AddGaussianSNR, TimeStretch, PitchShift self.augment = Compose([ AddGaussianNoise(min_amplitude=0.005, max_amplitude=0.015, p=poss), AddGaussianSNR(min_snr_in_db=5.0, max_snr_in_db=40.0, p=poss), TimeStretch(min_rate=0.8, max_rate=1.2, p=poss), PitchShift(min_semitones=-2, max_semitones=2, p=poss) ])These four augmentation methods are enough for current training. But thePitchShiftmet ...继续阅读 (99)

Just write this snippet for my practice of 0-1 Knapsack Problem:values = [1, 2, 3, 4, 5] weights = [3, 2, 1, 9, 6] max_weight = 12 def knappack(): n = len(values) dp = [[0] * (max_weight+1) for _ in range(n)] print(dp) for i in range(n): for j in range(1, max_weight+1): if weights[i] > j: dp[i][j] = dp[i-1][j] else: dp[i][j] = max(dp[i-1][j], dp[i-1][j-weights[i]] + values[i]) print(dp[-1][-1]) knappack() ...继续阅读 (92)

After reading theclassic paperabout Skip-list, I tried to implement it by myself. But then I found a line of pseudo-code in the “Delete” function that I couldn’t understand:Seems all the elements in “update” will point to “x”, so why do we need to check here? Maybe I can ignore this checking. Now here comes a part of my code:def erase(self, num: int) -> bool: update = Node(-1) curr = self._head for level in range(MAX_LEVEL-1, -1, -1): while curr._forward[level] and curr._forward[level]._key < num: ...继续阅读 (100)

This paperintroduced a method to extract only segments with bird sound from an audio file. Since the paper didn’t give any code, I started to write it by myself.Here is the Python implementation:import cv2 import time import torch import librosa import soundfile as sf import numpy as np from torchlibrosa.stft import LogmelFilterBank, Spectrogram class CFG: n_fft = 2048 hop_length = 512 sample_rate = 32000 n_mels = 64 fmin = 150 fmax = 150000 class SignalExtractor: def __init__(self): self.spectrogram_extractor = Spectrogram( n_fft=CFG.n_ ...继续阅读 (102)

Trying to test theIntel extension for PyTorchin my project, but it reported errors:Traceback (most recent call last): File "reviewjpgs_optimaztion_testing.py", line 27, inimport intel_extension_for_pytorch as ipex ...继续阅读 (90)

I created a simple Argo job to pull messages from a Google Cloud Pub/Sub topic. Permission has been given to the service account of GKE’s workload identity. But the Argo job failed with errors:argo submit example.json -n argoprojhello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable hello-world-pqbm5: return callable_(*args, **kwargs) hello-world-pqbm5: File "/usr/local/lib/python3.9/dist-packages/grpc/_channel.py", line 1030, in __call__ hello-world-pqbm5: return _end_unary_response_blocking(state ...继续阅读 (93)