Thermal energy harvesting using thermoelectrochemical cells (thermocells) is a sustainable method to produce electricity without carbon dioxide emissions. The solvent and redox couple used in the electrolyte play an important role in determining both the safety and performance of thermocells, and development of leak-free electrolytes with high performance is particularly important for transportable devices. Here, the application of aqueous and non-aqueous electrolytes containing the [Co(bpy)]2+/3+ redox couple in both liquid and solid forms was studied. Cellulose was used as an environmentally friendly material for solidification of the different liquid electrolytes. The properties and performance of the new aqueous [Co(bpy)]2+/3+ electrolytes was compared to those containing the Fe(CN)63−/4− couple, both in liquid and quasi-solid state electrolytes. Higher diffusivity for the cobalt redox ions was observed in the aqueous electrolyte compared to the non-aqueous electrolytes, while the Seebeck coefficient of the redox couple, which determines the open circuit voltage of the thermocell, was largest in the organic solvents. No significant effect of solidification on the Seebeck coefficient was observed.