Author: yodi

Categories
Tensorflow

Install Tensorflow 1 with RTX 4090 or CUDA 11 and 12

There is a quick way to install and run Tensorflow version 1 (eg: Tensorflow 1.5) in Ubuntu, RTX 4090 and CUDA 12. Here are quick step to do it.

  1. Install Python 3.6 or Python 3.8

Depending on your needs, you can setup this using anaconda 

conda create -n 36 python=3.6
conda activate 36

2. Install NVIDIA python index

pip install --user nvidia-pyindex

3. Install Tensorflow 1.5 both GPU and CPU support

pip install --user nvidia-tensorflow[horovod]

Voila! Now you can run Tensorflow 1 using CUDA 11 or 12 on Ubuntu without problem.

Another alternative is using NVIDIA NGC.

Categories
Ubuntu

Adjust GPU Fan Speed NVIDIA on Ubuntu Server Headless

The problem when adjusting my dual RTX 4090 in Ubuntu Server 23.10 is, when running nvidia-settings its trigger error

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.

I’ve been search on internet and not found a better solution, then I decided try to install X and attached, while running nvidia-setting. The result its works. Here are the step by step on how to enable GPU fan speed in Ubuntu Server 23.10

  1. Install XServer
sudo apt install -y xorg xinit

2. Give permission to run X for user

Edit the file and change console into anybody to give permission to running X

sudo vim /etc/X11/Xwrapper.config

If you don’t change this, it will trigger error 

/usr/lib/xorg/Xorg.wrap: Only console users are allowed to run the X server
Continue reading “Adjust GPU Fan Speed NVIDIA on Ubuntu Server Headless”
Categories
Ubuntu

Fix Chrome Star Auto Bookmark

# Update 27 Feb 2024

The Chrome flags to disable simplified bookmarks is gone in the latest version. If you are using Ubuntu, you can downgrade chrome or chromium in Snap

Here are the steps

  1. Revert chromium version
snap revert chromium

This will be rolling back to version Version 121.0.6167.160 (Official Build) snap (64-bit)

Which now, when you access chrome://flags it will have bookmark-flow features

2. Disable auto-update features

snap refresh --hold=forever

or

snap refresh --hold=<duration> <snap1> <snap2>

This is annoying where the bookmark or star is automatically added. To remove this feature

chrome://flags/#simplified-bookmark-save-flow

The disable it!

Categories
ML

Fix PySpark contains a task of very large size. The maximum recommended task size is 1000 KiB

If you got this error when running your notebook with warning

contains a task of very large size. The maximum recommended task size is 1000 KiB

This mean PySpark warning you to increase the partition or parallelism (and might memory as well).

Example code to configure it, where you can adjust based on your workstation memory. In my case, is 192GB is my max memory

import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--driver-memory 192g --executor-memory 16g pyspark-shell'

# add this one in your spark configuration
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

Full implementation

import os
from pyspark import SparkContext

os.environ['PYSPARK_SUBMIT_ARGS'] = '--driver-memory 192g --executor-memory 16g --executor-cores 10 pyspark-shell'
os.environ['PYARROW_IGNORE_TIMEZONE'] = '1'

builder = SparkSession.builder
builder = builder.config("spark.driver.maxResultSize", "5G")

spark = builder.master("local[*]").appName("FMClassifier_MovieLens").getOrCreate()
spark.conf.set("spark.sql.analyzer.failAmbiguousSelfJoin", "false")
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

Additional bonus description if you would like to increase the instances as well


spark = SparkSession.builder.config('spark.executor.instances', 4).getOrCreate()
spark.conf.set("spark.sql.analyzer.failAmbiguousSelfJoin", "false")
Categories
Ubuntu

Install CUDA 11 on Ubuntu 23.10

To solve Driver or CUDA 11 installation error in Ubuntu 23.10, the answer is to ensure its using the compatible GCC version. By default installation, it will using GCC 13 which is not working when compiling CUDA or NVIDIA Drivers (required GCC 10). Installing CUDA 11 is important to run Tensorflow that haven’t fully adapted with CUDA 12.

Failed to verify gcc version. See log at /var/log/cuda-installer.log for details.

First step to fixing this problem is to uninstall any nvidia and cuda installation made previously

sudo apt autoremove cuda* nvidia* --purge

Next, install GCC 10

MAX_GCC_VERSION=10
sudo apt install gcc-$MAX_GCC_VERSION g++-$MAX_GCC_VERSION
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-$MAX_GCC_VERSION $MAX_GCC_VERSION

Next, choose the GCC 10 as default by running this command

sudo update-alternatives --config gcc

Now, you all set! You can start to do installation of CUDA 11 in Ubuntu 23.10. Make sure to un-check the driver installation part (where we will install it later)

sudo ./cuda_11.8.0_520.61.05_linux.run

Next, I’m using Ubuntu NVIDIA default installation. So, I revert back the GCC to version 13, using the same command

sudo update-alternatives --config gcc
sudo apt install nvidia-driver-525

You can repeat the process like CUDNN, TensorRT and others installation following my previous article here

Install Transformers Pytorch Tensorflow Ubuntu 2023

Finally, make sure if anything broken with NVCC, is to switch the GCC version to 10, not 13.

Categories
Ubuntu

Fix VSCode open Large file by increase Memory

When opening Netflix data around 1GB, the VSCode is crashed. My memory are pretty much 30% usage and have plenty room to open this 1GB file.

To fix this, either run from terminal

code --max-memory=12288mb

Or right click the menu in Ubuntu, and replace the launcher with this.

code  --max-memory=12288mb --unity-launch %F
Categories
ML

Fix Failed to load implementation from:dev.ludovic.netlib.blas.VectorBLAS

When you got error “Failed to load implementation from:dev.ludovic.netlib.blas.VectorBLAS” when running ALS training, the quickfix for Intel MKL are 

sudo ln -s /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_rt.so /usr/local/lib/libblas.so.3
sudo ln -s /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_rt.so /usr/local/lib/liblapack.so.3

Or follow this: https://spark.apache.org/docs/latest/ml-linalg-guide.html

Categories
Tensorflow

Solve No builder could be found in the director tensorflow dataset

When downloading movielens dataset from Tensorflow dataset, I got this error 

No builder could be found in the director tensorflow dataset

The quick solution is to upgrade 

pip install --upgrade tfds-nightly
Categories
Machine Learning

Create Custom Metric Accuracy for Trainer Classification

Here are a simple custom metric to measure accuracy, which you can attach it into Trainer parameter

from transformers import Trainer
from evaluate import load

metric = load('accuracy')

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    accuracy = metric.compute(predictions=preds, references=labels)

    return {'accuracy': accuracy}
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['validation'],
    data_collator=data_collator,
    compute_metrics=compute_metrics, <----- HERE
    tokenizer=tokenizer
)

trainer.train()
Categories
Ubuntu

Install WordPress in Ubuntu 22.04 LightSail AWS

Here are technical quick steps on setup a brand new WordPress website using NGINX, PHP-FPM, LetsEncrypt and MariaDB. After following the step, you will have a website running in very cheap LightSail AWS ~ $5 / month.

Lets follow this 7 steps and just require less than 5 minutes!

  1. Go to AWS Lightsail and launch a new instance.
    Go to https://lightsail.aws.amazon.com/ls/webapp/home/instances and launch a new instance. Create a static IP Address and attach into this instance. Then, you can assign the NS of domain into new IP address both www and non-www.

    Then go to networking and enable “443” port to ensure HTTPS allowed from firewall.
Continue reading “Install WordPress in Ubuntu 22.04 LightSail AWS”