Category: Machine Learning

Categories
Machine Learning

How to uninstall Cuda and replace with new version

The quickfix on how to uninstall current Cuda installed in Ubuntu not via software packages is using the uninstaller. For instance, I use cuda 11.8 and I need to downgrade it into 11.6.

So, I need to find the path and trigger this command

sudo /usr/local/cuda-11.8/bin/cuda-uninstaller

Last, we can clean-up entire cuda folder

sudo rm -rf /usr/local/cuda

If you have issue with GCC for the installed CUDA and need to downgrade or upgrade it, you can follow this

MAX_GCC_VERSION=11
sudo apt install gcc-$MAX_GCC_VERSION g++-$MAX_GCC_VERSION
sudo ln -s /usr/bin/gcc-$MAX_GCC_VERSION /usr/local/cuda/bin/gcc 
sudo ln -s /usr/bin/g++-$MAX_GCC_VERSION /usr/local/cuda/bin/g++
Categories
Machine Learning

Solve unsupported GNU version! gcc versions later than 11 are not supported!

When installing Python module like AutoGPTQ, you may got this errors

/usr/local/cuda/include/crt/host_config.h:132:2: error: #error -- unsupported GNU version! gcc versions later than 11 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.
        132 | #error -- unsupported GNU version! gcc versions later than 11 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.
            |  ^~~~~
      error: command '/usr/local/cuda/bin/nvcc' failed with exit code 1
      [end of output]

To solve this, we need to install GCC as following the maximum version

MAX_GCC_VERSION=11

sudo apt install gcc-$MAX_GCC_VERSION g++-$MAX_GCC_VERSION
sudo ln -s /usr/bin/gcc-11 /usr/local/cuda/bin/
Categories
Machine Learning

Install Transformers Pytorch Tensorflow Ubuntu 2023

To install transformers, Pytorch and Tensorflow works with GPU for the latest Ubuntu, several steps are required. This is how I successfully setup it and running several models with it.

Please make sure to install the latest NVIDIA drivers. I use RTX 4090 in this case. This is the link https://www.nvidia.com/download/driverResults.aspx/200481/en-us/

If you are using nouveau, you can disable it via

sudo bash -c "echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf"
sudo bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf"

sudo update-initramfs -u
sudo reboot
Continue reading “Install Transformers Pytorch Tensorflow Ubuntu 2023”
Categories
Machine Learning

Install Stable Difussion Automatic111, Torch 2.0 and Fix RTX 4090 Performance

I use a clean installation of Ubuntu 23.04 Lunar Lobster and Nvidia driver 525. If you already have the driver installed, here are the steps to improve Automatic111 Stable Diffusion performance to 40-44 it/s

  1. Install required Anaconda

Ubuntu 23.04 default Python version is 3.11 version. In this case, I will using Anaconda to provide Python 3.10. Download Anaconda and install

chmod a+x /Anaconda3-2023.03-1-Linux-x86_64.sh
./Anaconda3-2023.03-1-Linux-x86_64.sh
Continue reading “Install Stable Difussion Automatic111, Torch 2.0 and Fix RTX 4090 Performance”
Categories
Machine Learning

Solve Pandas Drop Duplicates still not unique in Value Counts

When using pandas drop duplicates, we may encountered rows that still have duplicating by checking via 

df.column_name.value_counts()

Not sure why Pandas drop duplicates performance showing inconsistent result. However, to remove duplicate row, produce 100% unique based on index or key column, you can use this

df_unique = df_unique.drop(df_unique[df_unique["key_column_name"].duplicated()].index)
df_unique.temp_id.value_counts()
Categories
Machine Learning

Install LightGBM use GPU in Linux Ubuntu

LightGBM can work faster in GPU. In PyCaret, I’m passing parameter use_gpu=True in TSForecastingExperiment() and got errors:

[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1
[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1
[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1

To enable this, we need to uninstall the current LightGBM and re-install the LightGBM with GPU. For Linux Ubuntu, its better to install pre-requisite packages

sudo apt install cmake build-essential libboost-all-dev

Make sure you already have Nvidia Toolkit installed

sudo apt install nvidia-cuda-toolkit

The first option, is installation inside conda environment

pip uninstall lightgbm -y

conda install -c conda-forge gcc=12.1.0
pip install lightgbm --config-settings=cmake.define.USE_GPU=ON --config-settings=cmake.define.OpenCL_INCLUDE_DIR="/usr/local/cuda/include/" --config-settings=cmake.define.OpenCL_LIBRARY="/usr/local/cuda/lib64/libOpenCL.so"

Second options, installation without environment

# Get LightGBM source.
git clone --recursive https://github.com/Microsoft/LightGBM.git
cd LightGBM/python-package/
# cmake specifying locations of OpenCL files.
sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..
# Compile.
sudo make
# Install for Python, using what we just compiled.
python setup.py install --precompile
Categories
Machine Learning

Upgrade AUTOMATIC1111 StableDiffusion Torch 2.0 and Xformers Linux

The default installation of Stable Diffusion Automatic111 using the old version of Torch 1.13.1, Torchvision 0.14, and related Xformers, the webui.sh will receive “No module ‘xformers’. Proceeding without it.”.

Pytorch 2.0 and Xformers are offering a big upgrade. Here are quick step on how to upgrade the Automatic111 web UI and load with xformers.

If you are using Anaconda, go open its terminal then go to stable-diffusion-webui/venv/bin and source activate to enter “venv” environment with python 3.10 as default.

Then, we can start to upgrade the torchvision, torch and xformers with this command

pip install torchvision --upgrade
pip install torch==2.0
pip install -U xformers

If receive an error like this, you can upgrade the torchvision version. 

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.14.1+cu117 requires torch==1.13.1, but you have torch 2.0.0 which is incompatible.
Successfully installed torch-2.0.0

After everything is done completely, you can fire-up your webui.sh along with xformers module loaded with command : “bash webui.sh –xformers”

Continue reading “Upgrade AUTOMATIC1111 StableDiffusion Torch 2.0 and Xformers Linux”
Categories
Machine Learning

Install Tensorflow run with GPU RTX in Windows 11

Note:

Please go to the latest update article here https://www.yodiw.com/fix-install-tensorflow-2-with-gpu-simple-version-2023/

Author

This tutorial will help you to run Tensorflow in Windows 11 to use GPU for modelling, with the latest Visual Studio, CUDA and latest drivers. In this case, I’m using RTX 3060Ti Lite Hashes and Python 3.7 – 3.10 for Windows. (I use Python 3.10.9)

TLDR;

We need to install

  • Clean Windows Installation and GPU drivers card
  • the Microsoft Visual C++ (MSVC) compiler 2022
  • the CUDA Toolkit 11.8
  • the cuDNN libraries
  • Install tensorflow

Pre-requisites

1. Windows fresh installation to remove the complexity of issues or bugs is recommended here. I did re-install my Windows 11 in this step

2. Instead of using Nvidia Geforce Experience, I uninstall the GPU drivers (back to Microsoft Basic) and use NVCleanstall to download my RTX 3060 drivers without others bloatware https://www.techpowerup.com/download/techpowerup-nvcleanstall/

3. I use Python 3.9 from Anaconda https://www.anaconda.com/products/distribution

Continue reading “Install Tensorflow run with GPU RTX in Windows 11”