How do we change the pitch of a cloned voice? Clone is too high pitched

[fury88]

I've take a long sample of a voice and the clone that comes out is too high pitched and doesn't really sound the same. I know the quality of the sample is important but also can we tweak the clone pitch?

[dosubot[bot]]

To change the pitch of a cloned voice, you can adjust the speed parameter in the base_speaker_tts.tts function. The speed parameter affects both the pitch and speed of the generated speech. Here is an example:

import os
import torch
from openvoice import se_extractor
from openvoice.api import BaseSpeakerTTS, ToneColorConverter

ckpt_base = 'checkpoints/base_speakers/EN'
ckpt_converter = 'checkpoints/converter'
device="cuda:0" if torch.cuda.is_available() else "cpu"
output_dir = 'outputs'

base_speaker_tts = BaseSpeakerTTS(f'{ckpt_base}/config.json', device=device)
base_speaker_tts.load_ckpt(f'{ckpt_base}/checkpoint.pth')

tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')

os.makedirs(output_dir, exist_ok=True)

source_se = torch.load(f'{ckpt_base}/en_default_se.pth').to(device)
reference_speaker = 'resources/example_reference.mp3'
target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter, target_dir='processed', vad=True)

save_path = f'{output_dir}/output_en_default.wav'

# Run the base speaker tts
text = "This audio is generated by OpenVoice."
src_path = f'{output_dir}/tmp.wav'
# Adjust the speed to change the pitch
speed = 0.8  # Decrease the speed to lower the pitch
base_speaker_tts.tts(text, src_path, speaker='default', language='English', speed=speed)

# Run the tone color converter
encode_message = "@MyShell"
tone_color_converter.convert(
    audio_src_path=src_path, 
    src_se=source_se, 
    tgt_se=target_se, 
    output_path=save_path,
    message=encode_message)

In this example, setting speed = 0.8 will lower the pitch of the cloned voice. Adjust the speed value as needed to achieve the desired pitch ^[1][2].

To continue talking to Dosu, mention @dosu.

[fury88]

@dosu, the example you have on the github homepage for checkpoints_v2 does not change the pitch, only the speed.

[dosubot[bot]]

To change the pitch of a cloned voice using the checkpoints_v2 example, you can adjust the speed parameter in the tts method of the BaseSpeakerTTS class. The speed parameter indirectly affects the pitch of the generated audio. Here is the relevant part of the code with the speed parameter highlighted:

import os
import torch
from openvoice import se_extractor
from openvoice.api import BaseSpeakerTTS, ToneColorConverter

ckpt_base = 'checkpoints/base_speakers/EN'
ckpt_converter = 'checkpoints/converter'
device="cuda:0" if torch.cuda.is_available() else "cpu"
output_dir = 'outputs'

base_speaker_tts = BaseSpeakerTTS(f'{ckpt_base}/config.json', device=device)
base_speaker_tts.load_ckpt(f'{ckpt_base}/checkpoint.pth')

tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')

os.makedirs(output_dir, exist_ok=True)

source_se = torch.load(f'{ckpt_base}/en_default_se.pth').to(device)
reference_speaker = 'resources/example_reference.mp3'
target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter, target_dir='processed', vad=True)

save_path = f'{output_dir}/output_en_default.wav'

# Adjust the speed to change the pitch
speed = 1.2  # Increase the speed to raise the pitch, decrease to lower the pitch

# Run the base speaker tts
text = "This audio is generated by OpenVoice."
src_path = f'{output_dir}/tmp.wav'
base_speaker_tts.tts(text, src_path, speaker='default', language='English', speed=speed)

# Run the tone color converter
encode_message = "@MyShell"
tone_color_converter.convert(
    audio_src_path=src_path, 
    src_se=source_se, 
    tgt_se=target_se, 
    output_path=save_path,
    message=encode_message)

By adjusting the speed parameter, you can control the pitch of the cloned voice. Increasing the speed value will raise the pitch, while decreasing it will lower the pitch ^[1][2].

To continue talking to Dosu, mention @dosu.

[fury88]

@dosu, no that does not work. sorry. The speed only changes time, not pitch.

[dosubot[bot]]

To adjust the pitch independently of the speed, you can use the librosa library, which is a dependency of OpenVoice. Here is an example of how you can use librosa to adjust the pitch of an audio signal:

import librosa
import numpy as np

# Load an audio file
y, sr = librosa.load('path_to_audio_file.wav')

# Adjust the pitch by n_steps (positive for higher pitch, negative for lower pitch)
n_steps = 4  # Example: raise pitch by 4 semitones
y_shifted = librosa.effects.pitch_shift(y, sr, n_steps=n_steps)

# Save the pitch-shifted audio
librosa.output.write_wav('path_to_pitch_shifted_audio.wav', y_shifted, sr)

This code snippet demonstrates how to load an audio file, adjust its pitch, and save the modified audio using librosa ^[1][2].

To continue talking to Dosu, mention @dosu.