Machine Learning

Install Transformers Pytorch Tensorflow Ubuntu 2023

To install transformers, Pytorch and Tensorflow works with GPU for the latest Ubuntu, several steps are required. This is how I successfully setup it and running several models with it.

Please make sure to install the latest NVIDIA drivers. I use RTX 4090 in this case. This is the link

If you are using nouveau, you can disable it via

sudo bash -c "echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf"
sudo bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf"

sudo update-initramfs -u
sudo reboot
Machine Learning

Install Stable Difussion Automatic111, Torch 2.0 and Fix RTX 4090 Performance

I use a clean installation of Ubuntu 23.04 Lunar Lobster and Nvidia driver 525. If you already have the driver installed, here are the steps to improve Automatic111 Stable Diffusion performance to 40-44 it/s

  1. Install required Anaconda

Ubuntu 23.04 default Python version is 3.11 version. In this case, I will using Anaconda to provide Python 3.10. Download Anaconda and install

chmod a+x /
Machine Learning

Solve Pandas Drop Duplicates still not unique in Value Counts

When using pandas drop duplicates, we may encountered rows that still have duplicating by checking via


Not sure why Pandas drop duplicates performance showing inconsistent result. However, to remove duplicate row, produce 100% unique based on index or key column, you can use this

df_unique = df_unique.drop(df_unique[df_unique["key_column_name"].duplicated()].index)
Machine Learning

Install LightGBM use GPU in Linux Ubuntu

LightGBM can work faster in GPU. In PyCaret, I’m passing parameter use_gpu=True in TSForecastingExperiment() and got errors:

[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1
[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1
[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1

To enable this, we need to uninstall the current LightGBM and re-install the LightGBM with GPU. For Linux Ubuntu, its better to install pre-requisite packages

sudo apt install cmake build-essential libboost-all-dev

Make sure you already have Nvidia Toolkit installed

sudo apt install nvidia-cuda-toolkit

The first option, is installation inside conda environment

pip uninstall lightgbm -y

conda install -c conda-forge gcc=12.1.0

pip install lightgbm --install-option=--gpu --install-option="--opencl-include-dir=/usr/local/cuda/include/" --install-option="--opencl-library=/usr/local/cuda/lib64/"

Second options, installation without environment

# Get LightGBM source.
git clone --recursive
cd LightGBM/python-package/
# cmake specifying locations of OpenCL files.
sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/ -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..
# Compile.
sudo make
# Install for Python, using what we just compiled.
python install --precompile

Install Stable Diffusion Windows and Fix RTX performance 2023

Many feedback about performance NVIDIA RTX after installing Stable Diffusion Automatic1111. I will explained a simple way to install and fix the RTX 4090 performance within 5 minutes

First, make sure you have Python 3.10 in your Windows. You can use Anaconda or native Python installation.

  1. Clone stable diffusion git repository to your local directory

2. Install Stable Diffusion with xformers

This part is tricky. By default, it will install Torch 2.1.0, however the latest xformers will required to use torch 2.0. Which later you will encountered the problems like :

AssertionError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check

The solution for installation both xformers and torch inside stable difussion is to pass the arguments in installation

./webui.bat --xformers

Fix Install Tensorflow 2 with GPU Simple Version 2023

Tensorflow running use GPU in Windows sometimes is difficult to do, where many articles not pointing exactly which Tensorflow version, NVIDIA drivers and other requirement needed to achieve it in Windows 11. Especially when you have NVIDIA RTX Graphic card like 4090 or 3090 or similar version.

I will help to explain on how to install it properly and make it run. Here are the steps to do:

1.Installing the Latest Anaconda.

Go to Anaconda website and download its community distribution. You can try to activate in your terminal windows with “conda activate” and you will enter your base.

If you are using powershelll, you can try “conda init powershell” to load the environment by default

2.Installing Microsoft Visual Studio 2022 (not to confuse with VSCode editor).

Choose the community version at and install Desktop Environment with C++


Install Numpy with OneAPI MKL for AMD in Ubuntu

NumPy uses libraries like BLAS, LAPACK, BLIS, or MKL to execute vector, matrix, and linear algebra operations. It’s acknowledged that Intel with MKL (Math Kernel Library) is quite more mature in this math operation than other libraries due to resources and experiences.

If you want to leverage Intel OneAPI MKL as backend for your Numpy, especially on Intel chip (or AMD if you want to try), here are a quick step for installation in Ubuntu (I use the latest ubuntu 23.04 Lunar Lobster).

First, download the required softwares

sudo apt install build-essential python3-pip python3 python3-dev libomp-dev

A context, libomp-dev will help you to avoid the error “Solve undefined symbol: omp_get_num_procs” when importing the numpy libraries in the python interpreter.

Second, we will install the Intel Base-kit:

Or follow the commands below

Machine Learning

Upgrade AUTOMATIC1111 StableDiffusion Torch 2.0 and Xformers Linux

The default installation of Stable Diffusion Automatic111 using the old version of Torch 1.13.1, Torchvision 0.14, and related Xformers, the will receive “No module ‘xformers’. Proceeding without it.”.

Pytorch 2.0 and Xformers are offering a big upgrade. Here are quick step on how to upgrade the Automatic111 web UI and load with xformers.

If you are using Anaconda, go open its terminal then go to stable-diffusion-webui/venv/bin and source activate to enter “venv” environment with python 3.10 as default.

Then, we can start to upgrade the torchvision, torch and xformers with this command

pip install torchvision --upgrade
pip install torch==2.0
pip install -U xformers

If receive an error like this, you can upgrade the torchvision version.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.14.1+cu117 requires torch==1.13.1, but you have torch 2.0.0 which is incompatible.
Successfully installed torch-2.0.0

After everything is done completely, you can fire-up your along with xformers module loaded with command : “bash –xformers”


Migrate WordPress to Upgraded Ubuntu AWS Server

I move WordPress from Ubuntu Server 21.04 to 23.04 with several easy steps. Here are the quick guideline on how to migrate WordPress safely without breaking

First, Backup all necessary data
Open your terminal and start with your home path ~/

sudo mysqldump -u root -p yourdb > db.sql

Second, Compress etc, home and any folder needed

sudo zip -r /etc
sudo zip -r /home/ubuntu

Transfer the file to the new server. If you need to use pem-key and using either windows or linux, use SCP to transfer pem-key from local computer to old-server. This is needed so you can transfer file from old to new server

scp -i .\your-key.pem -r .\your-key.pem ubuntu@your-ec2-server-ip:/home/ubuntu

Now your key pem is exists in home path in old server. Its time to transfer the whole zip files backup to the new server home path

scp -i your-key.pem *.zip ubuntu@your-new-server-EC2:/home/ubuntu/

Once its done, you just need to extract the folder and replace whatever needed configuration and databases to the new server.

If you encountered by permission denied (13) on NGINX,

/php/php8.0-fpm.sock:", host: "www. com"
2023/04/23 16:31:33 [crit] 17224#17224: *28 stat() "/home/ubuntu/" failed (13: Permission denied), client:, server: .com, request: "GET / HTTP/1.1", host: "www. .com"
2023/04/23 16:31:33 [crit] 17224#17224: *28 stat() "/home/ubuntu/" failed (13: Permission denied), client:, server: .com, request: "GET / HTTP/1.1", host: ""
2023/04/23 16:31:33 [error] 17224#17224: *28 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 182.25352.107, server: .com, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.0-fpm.sock:", host: ""

the quickfix solution

sudo gpasswd -a www-data ubuntu
sudo service php8.1-fpm restart
sudo service nginx restart

Done! Now you can smoothly migrate WordPress from the old ubuntu server to the new ubuntu server without problem. The last step is to change the IP-address in NS. If you are using AWS Route 53, just change the IP address in the record and you are ready to go!


Solve Profile PS1 cannot be loaded UnauthorizedAccess

If you have this error when opening CMD, Powershell or terminal

WindowsPowerShell\profile.ps1 cannot be loaded because running scripts is disabled on
this system. For more information, see about_Execution_Policies at https:/
At line:1 char:3
+ . 'C:\Users\dev\Documents\WindowsPowerShell\profile.ps1'
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : SecurityError: (:) [], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess

The quick fix is open “Powershell” with administrator access and execute this command

Set-ExecutionPolicy Unrestricted